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1.  EXECUTIVE  SUMMARY 


The  Vehicle-Pilot  Integration  Branch  of  Wright  Laboratory  (WL/FIGP)  conducted 
a  pilot-in  the-loop  study  to  assess  the  utility  of  a  prototype  decision  aid  to  support  the 
pilot’s  use  of  electronic  approach  plates  in  flying  instrument  approaches.  The 
prototype  decision  aid  system  that  was  evaluated  is  the  Approach  Procedures  Expert 
System  (APES). 

Sixteen  pilots  flew  a  series  of  instrument  approaches  in  a  cockpit  simulator.  The 
presence  or  absence  of  the  decision  aid,  the  orientation  of  the  electronic  approach  plate 
(EAP)  and  task  difficulty  were  varied  across  testing  sessions.  Two  EAP  orientations 
were  investigated:  North-Up  and  Track-Up.  Task  difficulty  was  implemented  at  two 
levels.  In  the  high  task  loading  condition,  pilots  flew  non-precision  approaches  with 
wind  gusts  incorporated  into  the  aeromodel.  They  also  had  no  prior  review  the 
approach.  In  the  low  task  loading  condition,  pilots  flew  precision  approaches  without 
wind  gusts  and  reviewed  the  approach  plate  prior  to  flying  the  approach. 

WL/FIGP  conducted  the  experiments  in  the  Microprocessor  Applications  for 
Graphics  and  Interactive  Communications  (MAGIC)  single-seat  fighter  simulator.  The 
simulator  cockpit  contained  a  force-control  stick  and  color  Cathode  Ray  Tube  (CRT) 
displays  that  depicted  head-down  flight  information,  the  EAP,  APES  textual  output, 
and  touch  screen  data  entry  information. 

Flight  performance,  workload,  situational  awareness,  and  questionnaire  data  were 
collected.  Analyses  of  Variance  (ANOVA)  and  the  appropriate  post-hoc  tests  were 
applied  to  the  performance,  workload,  and  situational  awareness  data.  For  the 
questionnaire  data,  frequency  distributions  and  means  were  calculated  and  comments 
were  summarized. 

The  results  showed  that  the  APES  enhanced  the  pilot’s  ability  to  perform 
instrument  approach  tasks  (Objective  1)  compared  to  flying  the  approaches  without 
APES  assistance.  With  APES  assistance,  pilots  deviated  less  from  assigned  altitudes. 
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especially  during  high  task  loading.  They  also  deviated  less  from  assigned  airspeed 
during  the  initial  and  final  approach  phases.  Pilots  rated  their  workload  lower  with 
APES,  particularly  during  high  task  loading,  and  their  situational  awareness  higher. 
Study  findings  also  indicate  that  APES  effectiveness  was  not  influenced  by  the 
electronic  approach  plate  orientation  (i.e.,  north-up  or  track-up). 

Regarding  the  performance  of  the  decision  aid  (Objective  2),  pilots  rated  APES’ 
logic,  consistency  and  timeliness  as  above  “moderately  acceptable,”  however,  some 
refinements  were  indicated.  Pilot  comments  indicated  that  the  deviation  tolerance 
windows  for  APES  voice  activation  were  too  stringent,  especially  the  airspeed  and 
altitude  voice  prompts.  Also  the  timeliness  of  the  procedural  prompts,  given  at  the 
approach  fixes,  were  reported  as  being  “hurried.”  In  general,  pilots  thought  that  APES 
would  improve  flight  safety,  however,  some  expressed  concern  with  the  consequences 
of  being  over-reliant  on  the  system. 

Pilots  rated  the  APES  PVl  as  “acceptable”  (Objective  3);  however,  refinements 
were  suggested.  Pilots  commented  that  the  phraseology  of  APES  voice  prompts 
should  be  more  specific,  particularly  during  the  final  phase  of  the  approach.  Pilots  also 
commented  that  the  PVl  should  allow  for  pilot  settings  and/or  adjustments  to  tolerance 
values  for  decision  aid  activation. 
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2.  INTRODUCTION 


This  report  describes  the  Approach  Procedure  Expert  System  (APES);  and  the 
testing  methodology  and  results  of  the  WL/FIGP  pilot-in-the-loop  evaluation  of  the 
APES.  Potential  areas  for  improvement  of  the  APES  and  recommendations  for  further 
research  are  also  discussed. 

2.1  Background 

The  approach  and  landing  phase  of  flight  is  considered  to  be  one  of  the  most 
workload  intensive.  In  fact,  recent  studies  have  shown  that  up  to  50%  of  civilian 
aircraft  accidents  occur  during  this  phase  (Blanchard,  1991).  A  major  factor 
contributing  to  these  incidents  is  the  extensive  cognitive  demand  placed  upon  the  pilot 
(Clay,  M.  and  Barlow,  T.,  1994).  The  pilot  must  recall  and  apply  specific  instrument 
flight  rules,  remember  correct  task  sequences,  and  calculate  timings,  while 
simultaneously  controlling  the  aircraft  and  monitoring  its  performance.  In  addition, 
the  pilot  must  integrate  information  from  multiple  sources  and  replan  according  to  air 
traffic  control’s  (ATC’s)  redirection.  Because  of  the  extensive  cognitive  load,  number 
of  procedures,  and  “rules  of  thumb”  associated  with  the  instrument  approach,  we 
believe  the  instrument  approach  domain  is  well-suited  for  a  decision  aid  application. 

Decision  aiding  has  been  used  to  help  pilots  perform  such  diverse  tasks  as  threat 
avoidance,  terrain  following,  mission  planning  and  re-routing,  and  flight  management. 
Decision  aids  produced  through  such  efforts  include  the  Pilot's  Associate  (Leavitt  and 
Smith,  1989),  the  Cockpit  Assistance  System  (Harrison,  Saunders,  and  Janowitz, 
1994),  the  Emergency  Procedures  Expert  System  (Harrison,  Saunders,  and  Janowitz, 
1994),  and  the  Flight-plan  Interactive  Negotiation  and  Decision- Aiding  System  for 
Enroute  Rerouting  (Bitterman  et  al,  1994).  In  general,  these  efforts  have  demonstrated 
the  potential  utility  of  a  computer-based  decision  aid  for  assisting  the  pilot  in  task 
accomplishment. 
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2,2  Study  Objectives 


The  goal  of  this  study  was  to  assess  the  usefulness  and  performance  of  a  prototype 
decision  aid  for  flying  instrument  approaches  by  evaluating  APES  in  a  pilot-in-the- 
loop  simulation.  Because  the  decision  aid  is  inseparable  from  its  interface,  both  the 
decision  aid  and  the  pilot-vehicle  interface  (PVI)  were  evaluated;  however,  the 
emphasis  of  the  study  was  placed  on  the  value  of  the  decision  aid’s  advice.  The 
objectives  of  the  study  were  to; 

1)  Assess  the  effectiveness  of  APES  for  supporting  approach  tasks  and  its  potential 
for  reducing  pilot  workload,  increasing  situational  awareness,  and  improving 
performance. 

2)  Assess  the  performance  of  the  decision  aid  to  determine  if  APES  advice  was 
accurate  and  timely  enough  to  assist  the  pilot  in  flying  instrument  approaches. 

3)  Assess  the  understandability  and  useability  of  the  pilot-vehicle  interface  to 
determine  if  the  interface  allowed  the  pilot  to  easily  interpret  and  use  APES  advice. 
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3.  APPROACH  PROCEDURES  EXPERT  SYSTEM  (APES) 


The  intent  of  the  APES  prototype  is  to  reduce  pilot  workload,  increase  situational 
awareness,  and  improve  performance  and  safety.  The  APES  simultaneously  monitors 
aircraft  performance,  informs  the  pilot  of  appropriate  corrective  actions  when 
deviations  occur,  and  provides  procedural  advice  (see  Table  1)  according  to  the  phase 
of  the  approach  (i.e.,  holding,  initial  approach,  final  approach,  missed  approach).  To 
accomplish  this,  the  APES  functions  in  two  assistant  roles:  as  an  “advisory  copilot” 
and  as  an  “advisory  pilot.”  As  an  “advisory  copilot”  the  decision  aid  advises  and 
prompts  the  pilot  as  a  copilot  would  in  a  multi-crew  environment,  such  as  advising 
when  the  aircraft  deviated  from  assigned  parameters  (e.g.,  altitude,  airspeed,  etc.).  As 
an  “advisory  pilot”  the  decision  aid  provides  guidance  relevant  to  the  instrument  flight 
rules  (IFRs)  needed  for  the  specific  phases  of  the  approach. 

Audio,  a  natural  form  of  communication  that  would  exist  between  the  pilot  and 
copilot,  is  the  primary  pilot-vehicle  interface  for  the  APES.  Visual  messages  are 
employed  for  redundancy  and  when  it  would  be  impractical  to  use  audio. 

The  following  sections  describe  the  developmental  process,  the  APES  system 
architecture,  and  the  Pilot-Vehicle  Interface  (PVI). 

3.1  Overview  of  the  APES  Developmental  Process 

The  first  step  in  the  development  of  the  APES  was  capturing  the  expertise  of 
experienced  pilots  through  a  knowledge  acquisition  process.  A  knowledge  engineer 
conducted  an  iterative  interview  process  with  several  subject  matter  experts.  This 
process  identified  the  precise  steps  that  were  necessary  for  flying  the  various  phases  of 
an  approach.  The  knowledge  engineer  then  modeled  the  actions  recommended  by  the 
expert  pilots  and  created  process  flow  diagrams.  The  process  flow  diagrams  served  as 
a  basis  for  the  APES  algorithm.  Representative  examples  of  these  process  flow 
diagrams  are  presented  in  Appendix  D. 
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The  APES  prototype  was  then  integrated  into  the  MAGIC  simulator  and  tested  in 
an  iterative  check-out  process.  Test  approaches  were  flown  with  various  flying 
patterns  to  exercise  all  of  APES  decision  points  and  to  determine  if  APES  was 
functioning  as  intended.  Design  flaws  were  identified  and  corrected.  Upon 
completion  of  the  check-out  process,  a  verification  test  was  conducted.  A  pilot, 
unfamiliar  with  the  APES,  flew  all  of  the  approaches  that  were  used  in  the  study. 
Design  deficiencies  that  went  undetected  during  the  iterative  design  process  were 
identified  and  corrected.  APES  was  then  formally  evaluated  in  the  current  study. 

3.2  APES  Architecture 

The  APES  prototype  system  consists  of  the  following  basic  components: 

1)  a  dynamic  aircraft  status  file 

2)  a  set  of  facts  representing  aircraft-specific  and  approach-specific  databases 

3)  a  set  of  rules  where  the  expert  knowledge  resides 

4)  a  forward-chaining  inference  engine  that  uses  the  Rete  algorithm  for 
deciding  which  rules  to  fire  based  on  the  facts. 

The  interaction  of  these  components  is  depicted  in  Figure  1. 


Figure  1.  APES  System  Architecture 
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3.2.1  APES  Inputs 


As  depicted  in  Figure  1,  input  to  the  APES  comes  from  three  sources:  current 
aircraft  flight  parameters,  a  database  of  aircraft  specific  facts,  and  a  database  of 
approach  specific  facts.  Examples  of  the  types  of  input  that  are  used  by  APES  include 
the  following 

•  Aircraft  Status  Data 

-  Current  Altitude  /  Heading  /  Airspeed 

-  Current  Navigation  Aid  Radial 

-  Current  Navigational  Radio  Channel 

•  Aircraft  Specific  Facts 

-  Holding  Airspeed 

-  Fuel  Weight 

-  Approach  Airspeed 

•  Approach  Specific  Facts 

-  Holding  Altitude 

-  Final  Approach  Course 

An  aircraft  specific  input  file  is  created  for  each  aircraft  type  in  order  to  allow  a 
generic  APES  to  be  embedded  in  aircraft  (or  aircraft  simulators)  of  different  types. 
Data  for  the  aircraft  are  loaded  from  the  corresponding  aircraft  specific  data  file  during 
program  initialization.  For  purposes  of  this  study,  the  aircraft  specific  facts  were 
limited  to  an  F-16  aircraft.  Also,  the  approach  specific  facts  were  limited  to  eight 
approach  plates,  however,  the  APES  can  accommodate  as  many  approach  plates  as 
computer  storage  availability  allows. 

Input  data  and  their  usage  within  the  APES  are  illustrated  in  the  process  flow 
diagrams  contained  in  Appendix  D. 

3.2.2  APES  Implementation 

The  expert  systems  development  tool  used  for  this  study  was  the  C  Language 
Integrated  Production  System  (CLIPS)  developed  at  the  NASA  Johnson  Space  Center 
(CLIPS,  1993).  CLIPS  is  a  distinguished  member  of  the  OPS-5  family  of  expert 
systems  shells,  and  has  been  extensively  used  in  many  applications,  including  a  variety 
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of  NASA  missions.  The  CLIPS  inference  engine  uses  the  highly  efficient  data-driven 
Rete  algorithm  (Forgy,  1982),  which  contributes  to  CLIPS’  excellent  run-time 
characteristics.  CLIPS  avoids  the  timing  problems  associated  with  slower  running 
expert  systems  because  the  Rete  algorithm  does  not  reconsider  a  rule  (that  has  already 
been  executed)  for  activation  until  a  subsequent  change  in  the  value  of  one  or  more  of 
its  antecedents  has  occurred. 

3.3  APES  Pilot-Vehicle  Interface 

The  primary  pilot  interface  for  the  APES  is  voice  message  presentation.  To 
output  a  voice  message,  the  APES  passes  a  text  string  to  the  voice  module  of  the 
Silicon  Graphics  host  system.  The  Silicon  Graphics  system  then  generates  the  voice 
message  by  combining  words  that  are  contained  in  a  vocabulary  database  of 
approximately  50  words. 

APES  voice  messages  are  reinforced  with  the  visual  presentation  of  text 
information.  The  APES  continually  displays  updated  target  values  for  radio 
channel/frequency,  altitude,  airspeed,  heading,  and  course  in  a  scratchpad  area  to  allow 
the  pilot  to  manipulate  appropriate  command  marker  and  course  indicator  settings. 
APES  also  displays  current  (target)  values  for  radio,  altitude,  airspeed,  heading  and 
course  on  a  dedicated  CRT.  This  CRT  is  also  used  to  display  more  complex  textual 
information,  such  as  pre-approach  and  final  approach  checklists.  The  cockpit  displays 
are  described  in  more  detail  in  Section  3.2  and  depicted  in  Figure  2. 

Table  1  shows  the  various  types  of  correction  advice  given  to  the  pilot  as  part  of 
the  APES  “advisory  copilof  ’  component.  As  shown,  this  advice  is  given  for  airspeed, 
altitude,  heading  and  course  aircraft  parameters,  with  each  parameter  having  a  preset 
tolerance  window  (determined  by  subject  matter  experts  and  pre-testing  simulation 
runs)  for  decision  aid  activation.  The  APES  advisory  copilot  is  only  activated  when 
the  aircraft  deviates  outside  a  parameter’s  tolerance  window.  When  this  occurs, 
APES  advises  the  pilot  of  the  target  value  and  provides  correction.  For  example,  if  the 
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pilot  deviates  more  than  +/-  2  degrees  from  course,  the  APES  “advisory  copilot” 
component  would  compute  a  heading  to  re-intercept  the  course  and  announce  “Turn 
(Right/Left)  Heading  XXX.”  Once  reestablished  on  the  course,  APES  would  then 
announce  “Maintain  Course  XXX.”  Target  course  is  also  displayed  in  the  scratchpad 
area  on  the  lower  CRT. 


Table  1.  PVI  Correction  Prompts 


“ADVISORY  COPILOT”  CORRECTION  PROMPTS 

DECISION  AID 
PARAMETER 

AUDIO  ADVICE 

VISUAL  ADVICE 

TOLERANCE 

WINDOW 

AIRSPEED 

“Maintain  XXX  knots“ 

Desired  airspeed  displayed  in  the 
scratch-  pad  for  command  airspeed 
marker  setting 

+/-  10  knots 

ALTITUDE 

“Maintain  altitude  XXX” 

Desired  altitude  displayed  in  the 
scratchpad  for  command  altitude 
marker  setting 

+/-  1 00  feet 

HEADING 

“Come  Right  (Left)  XXX  degrees” 
“Turn  (right/Left)  Heading  XXX” 
“Maintain  Heading  XXX” 

Desired  heading  displayed  in  the 
scratchpad  for  setting  of  the  heading 
“bug” 

+/-  5  degrees 

COURSE 

“Turn  (Right/Left)Heading  XXX 
“Maintain  Course  XXX“ 

Desired  course  displayed  in  the 
scratchpad  for  setting  of  the  course 
on  the  HSI 

+/-  2  degrees 

Table  2  depicts  the  various  types  of  procedural  advice  of  the  APES  “advisory 
pilot”  component.  The  procedural  advice  is  based  on  instrument  procedures 
documented  in  Air  Force  Manual  51-37  (1986)  and  on  specific  requirements  of  the 
approach  (which  are  displayed  on  the  electronic  approach  plate).  Unlike  the  “advisory 
copilot,”  the  “advisory  pilot”  component  will  always  give  procedural  advice  to  the 
pilots.  This  advice  differs  according  to  the  approach  and  phase  of  flight  (i.e.,  holding, 
initial  approach,  final  approach,  missed  approach).  For  example,  when  the  aircraft  is 
approximately  0.5  mile  from  the  Initial  Approach  Fix  (lAF),  the  APES  “advisory  pilot” 
component  would  compute  a  heading  and  announce  “Turn  (Right  /Left)  Heading 
XXX”  to  intercept  the  initial  approach  course  of  that  approach.  The  APES  would  also 
inform  the  pilot  to  descend  to  the  target  altitude  and  periodically  remind  the  pilot  to 
continue  descent  to  the  initial  approach  altitude. 
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Table  2.  PVI  Procedural  Prompts 


“ADVISORY  PILOT”  PROCEDURAL  PROMPTS 

DECISION  AID 
PARAMETER 

AUDITORY 

VISUAL 

TURN 

“Turn  (left/right)  Heading 
(XXX)” 

Textual  display  of  direction  appeared  on  the 
right  CRT  and  in  the  scratchpad. 

AIRSPEED 

“Set  Airspeed  XXX  knots” 
“Maintain  Airspeed  XXX 
knots.” 

Displayed  desired  airspeed  on  the  right  CRT  and 
in  the  scratchpad.  Reversed  video  highlight 
Airspeed  Select  Button  on  lower  CRT  to  cue  for 
quick  entry  of  command  value. 

ALTITUDE 

“XXX  feet  (above/below)” 
“Minimum  Altitude  XXX 
feet” 

“Begin  (Descent/Climb)” 

Displayed  desired  altitude  on  right  CRT  and  in 
scratch  pad.  Reversed  video  highlight  Altitude 
Select  Button  on  lower  CRT  to  cue  for  quick 
entry  of  command  value. 

NAVAID 

“Set  Navaid  Channel  and  / 
or  Frequency  (XXX  / 
XXX.XX)” 

Displayed  desired  freq./ch  on  right  CRT  and  in 
scratch  pad.  Reversed  video  highlight  Navaid 
Select  Button  on  lower  CRT  to  cue  for  quick 
entry  of  Navaid  Channel  and/or  Frequency. 

COURSE 

“Set  Inbound/Outbound 

Course  (XXX)” 

Displayed  desired  course  on  right  CRT  and  in 
scratch  pad.  Reversed  video  highlight  Course 
Select  Button  on  lower  CRT  to  cue  HSI  course 
entry. 
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4.  METHODOLOGY 


This  section  discusses  the  testing  methodology  that  was  used  to  accomplish  the 
pilot-in-the-loop  evaluation  of  the  APES.  Sixteen  pilots  flew  a  series  of  instrument 
approaches  in  a  cockpit  simulator.  The  presence  of  the  decision  aid,  the  orientation  of 
the  electronic  approach  plate  (EAP)  and  task  difficulty  were  varied  across  approaches. 
Task  difficulty  was  implemented  at  two  levels,  high  task  loading  and  low  tasking 
loading,  to  determine  the  benefits  of  the  APES  in  both  task  environments.  Two  EAP 
orientations  were  also  investigated,  North-Up  and  Track-Up,  to  determine  if  the  utility 
of  the  APES  would  vary  with  EAP  orientation. 

4.1  Background  and  Experience  of  Subject  Pilots 

Sixteen  volunteer  pilots  participated  in  the  study,  with  experience  that  ranged 
from  first  pilot  to  evaluator  pilot  and  an  average  total  flying  time  of  2448  hours.  One 
half  of  the  pilots  flew  multi-place  transport  aircraft  and  the  other  half  flew  fighter-type 
aircraft.  Table  3  shows  the  background  and  experience  of  the  pilots  used  in  the  study. 


Table  3.  Background  and  Experience  of  Subject  Pilots 


SUBJECT 

PRIMARY 

AIRCRAFT 

TOTAL 
FLYING  TIME 

ACTIVE  FLYING  STATUS 
WITHIN  LAST  12  MONTHS 

1 

C-141 

2300 

YES 

2 

P-3 

3200 

YES 

3 

C-130 

3000 

NO 

4 

C-5 

2900 

NO 

5 

A-10 

1900 

NO 

6 

E-3A 

3500 

YES 

7 

F-1 1 1  A/E 

2700 

YES 

8 

F-15 

2400 

YES 

9 

C-20A 

4400 

YES 

10 

c-141 

4575 

YES 

11 

F-1 11 

2800 

NO 

12 

2500 

YES 

13 

F-16 

1000 

YES 

14 

F-1 6 

2000 

NO 

15 

F-16 

1750 

YES 

16 

F-16 

600 

YES 
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4.2  The  MAGIC  Simulator 


The  APES  study  was  conducted  in  the  MAGIC  simulator  cockpit.  The  single-seat 
fighter  cockpit  contained  five  color  CRT  displays  (only  four  of  which  were  used),  three 
banks  of  programmable  switches,  an  A-7  throttle,  and  a  force-control  stick.  Two 
speakers  were  located  behind  the  cockpit  seat  for  announcing  the  APES  audio  advice. 
Figure  2  shows  a  layout  of  the  cockpit.  An  F-16  aeromodel  was  used  to  drive  the 
simulator. 


The  center  CRT  was  sectioned  into  two  6x8  inch  areas.  The  left  side  was  used  to 
display  the  Electronic  Approach  Plate  (EAP)  formats.  The  right  side  was  used  to 
display  Attitude  Director  Indicator  (ADI)  and  Horizontal  Situation  Indicator  (HSI) 
flight  information.  A  round  dial  airspeed  indicator  and  altimeter  were  located  on  the 
respective  left  and  right  sides  of  the  ADI.  The  right  CRT  displayed  textual  output 
from  the  APES.  The  left  CRT  displayed  crew  alerting  status  and  a  tachometer. 
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The  lower  CRT  displayed  a  touch-sensitive  keypad  and  switches  for  entering 
navigation  radio  frequency/channel,  course,  airspeed,  altitude  and  heading  values. 
This  CRT  also  displayed  a  scratchpad  area  that  presented  APES  prompts  of  target 
values.  These  values  could  then  be  displayed  as  commanded  values  on  the  flight 
displays  by  pressing  “enter”  on  the  keyboard. 

For  this  study,  pilots  only  used  the  first  three  programmable  switches  below  the 
lower  center  monitor  and  the  first  two  programmable  switches  below  the  left  monitor. 
The  switches  below  the  center  monitor  were  used  to  select  the  approach  phase.  The 
first  switch,  labeled  Hold,  informed  APES  that  holding  was  required  prior  to  initiating 
the  approach.  The  second  switch,  labeled  lAF,  informed  APES  that  approach 
clearance  had  been  received  and  to  proceed  to  the  Initial  Approach  Fix.  The  third 
switch,  labeled  FAF,  informed  APES  that  vectors  to  final  had  been  received  to  proceed 
to  the  Final  Approach  Fix.  The  first  two  switches  below  the  left  monitor  were  used  to 
select  TACAN  and  VOR/ILS  navigation  radios,  respectively. 

4.2.1  Electronic  Approach  Plate  Formats 

The  Electronic  Approach  Plates  (EAPs)  were  electronic  depictions  of  the  paper 
approach  plates  that  pilots  use  for  flying  instrument  approaches.  Because  EAPs  are 
integrated  with  the  aircraft’s  navigational  system,  they  also  displayed  current  aircraft 
position  with  respect  to  the  approach  plate.  The  EAPs  were  developed  using  a  vector 
product  format  (MIL-STD-600006),  which  is  a  Geographic  Informational  System  data 
format  created  for  transmitting  geographical  digital  databases. 

Aircraft  position  was  presented  in  two  map  orientations  on  the  EAP.  In  the 
North-Up  orientation,  north  was  always  located  at  the  top  of  the  display  and  the 
aircraft  symbol  was  fixed  in  the  center  of  the  display  with  the  EAP  moving  underneath 
it  to  reflect  current  position.  The  aircraft  symbol  also  rotated  to  reflect  the  correct 
heading.  In  the  Track-Up  orientation,  the  fixed  aircraft’s  symbol  was  pointed  toward 
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the  top  of  the  display  to  reflect  the  aircraft’s  current  track  and  the  EAP  moved 
underneath  it  to  reflect  the  aircraft’s  position. 


A  continuous  zoom  function,  operated  by  a  thumb  switch  on  the  throttle,  was 
provided  to  improve  readability  on  the  EAP  vertical  or  plan  view  sections  (Liggett, 
1996).  The  section  to  be  zoomed  (i.e.,  plan  view  or  vertical  view)  was  selected  by 
touch;  however,  the  zoom  center  was  always  the  aircraft  position.  Figure  3  illustrates  a 
typical  EAP  that  was  used  in  the  study. 


Plan  View 
of  Approach 


Figure  3.  A  Typical  Electronic  Approach  Plate  (EAP) 


4.3  Training 

Training  consisted  of  two  sessions:  one  hour  of  ground  training  and  one  hour  of 
simulator  flight  training.  The  ground  training  consisted  of  a  standardized  briefing 
covering  the  purpose  of  the  study,  aeromodel  characteristics,  control  use,  cockpit 
layout,  description  of  basic  layout  of  EAP  formats,  and  experimental  procedures. 
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The  simulator  training  familiarized  the  pilots  with  all  aspects  of  the  test  scenario 
and  study  procedures.  Pilots  flew  two  practice  approaches  that  were  similar  to  the 
approaches  they  would  fly  in  the  data  collection  sessions.  The  first  practice  approach 
consisted  of  a  precision  approach,  without  decision  aiding,  to  familiarize  the  pilot  with 
the  aeromodel  and  the  EAP  in  a  Track-Up  orientation.  The  second  practice  approach 
consisted  of  a  non-precision  approach  (no  glideslope  information)  with  decision 
aiding,  in  a  North-Up  orientation.  This  was  to  familiarize  the  pilot  with  the  decision 
aid  and  the  EAP  in  a  North-Up  Orientation,  as  well  as  to  allow  more  practice  with  the 
aeromodel.  The  two  practice  trials  also  familiarized  the  pilots  with  both  the  precision 
and  non-precision  types  of  approach  which  were  used,  in  part,  to  manipulate  task 
difficulty. 

4.4  The  Approach  Instrument  Procedure  Task 

Each  pilot  flew  eight  approaches:  four  with  decision  aiding  (Aid)  and  four 
without  decision  aiding  (NoAid).  There  were  four  different  approaches:  one  approach 
contained  a  short  arc  with  a  west  runway,  a  second  approach  contained  a  short  arc  with 
a  south  runway,  a  third  approach  had  a  radial  approach  with  a  west  runway,  and  a 
fourth  had  a  radial  approach  with  a  south  runway.  The  other  four  approaches  were 
comparable  to  the  four  original  approaches  with  minor  changes,  such  as  different 
altitude  restrictions  and  Navaid  frequencies.  South  runways  were  included  to  examine 
the  complications  of  a  North-Up  orientation  (e.g.,  left  turns  on  the  EAP  could  be 
misinterpreted  as  right  turns). 

For  the  low  task  loading  condition,  pilots  flew  precision  approaches  that  gave  both 
localizer  and  glideslope  deviation  and  guidance  information.  They  were  also  allowed 
to  review  the  approach  before  flying  it.  For  the  high  task  loading  condition,  pilots  flew 
non-precision  approaches  and  were  provided  with  only  localizer  deviation  and 
guidance  information.  Also  pilots  did  not  review  the  approach  and  a  gust  model  was 
incorporated  into  the  aeromodel  to  induce  the  effect  of  turbulence. 
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The  simulation  started  with  the  aircraft  flying  over  the  navigational  aid  serving  the 
airfield,  in  the  general  direction  of  the  initial  approach  fix,  and  at  the  assigned  altitude 
and  airspeed.  The  navigation  radio  was  initially  tuned  to  the  appropriate  navaid, 
however,  the  pilot  was  required  to  make  all  subsequent  frequency  changes.  Position 
reports  and  air  traffic  control  were  not  required.  Pilots  were  cleared  to  maneuver  as 
required  within  the  guidelines  of  the  Federal  Aviation  Administration  Instrument 
Flight  Rules  (IFRs)  and  approach  specific  restrictions. 

For  each  approach,  the  pilot  flew  one  turn  in  holding  and  proceeded  to  the  initial 
approach  fix.  The  pilot  flew  the  initial  approach  track  to  the  final  approach  fix,  flew 
the  final  approach  to  the  missed  approach  point,  and  then  executed  missed  approach 
procedures.  The  testing  session  was  completed  after  the  missed  approach  procedures 
were  executed.  Each  approach  session  took  approximately  25-30  minutes  to  complete, 
for  approximately  4  hours  for  the  eight  approaches. 

The  following  performance  measures  were  collected  at  5  Hertz  during  each 
instrument  approach  task:  Root-Mean-Square  (RMS)  altitude  deviations,  RMS 
airspeed  deviations,  RMS  glideslope  deviations  (precision  approaches),  RMS  heading 
deviations,  and  RMS  course  deviations.  These  were  calculated  separately  for  the 
following  approach  phases:  holding  (holding  fix  to  initial  approach  fix),  initial 
approach  (initial  approach  fix  to  final  approach  fix),  and  final  approach  (final  approach 
fix  to  missed  approach  point). 

4.5  Questionnaires 

After  completing  each  approach,  the  pilot  was  asked  to  rate  the  ease  of  performing 
basic  approach  tasks  (Figure  4)  and  to  give  initial  comments  in  a  session  questionnaire. 
Upon  completion  of  all  eight  approaches,  the  pilot  completed  a  final  questionnaire. 
The  final  questionnaire  asked  for  pilot  ratings  on  the  logic,  consistency  and  timeliness 
of  APES  advice  and  its  ability  to  support  specific  approach  tasks.  The  final 
questionnaire  also  included  questions  concerning  the  understandability  and  useability 
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of  the  PVI.  The  questionnaires  allowed  a  detailed  assessment  of  the  APES  so  that 
potential  deficiencies  in  the  “advisory  copilot”  and  “advisory  pilot”  components  could 
be  identified. 


In  conjunction  with  the  final  questionnaire,  pilots  also  rated  workload  and 
situational  awareness  using  the  Subjective  Workload  Dominance  (SWORD)  technique 
(Vidulich,  1989).  A  SWORD  form  was  used  to  collect  pairwise  comparisons  between 
combinations  of  three  test  conditions:  Decision  Aid  (Aid  /  No  Aid),  Approach  Type 
(Non-Precision,  EAP  Review,  NoWind  /  Precision,  NoEAP  Review,  Wind)  and  EAP 
Orientation  (North-Up  /  Track-Up)  The  comparisons  were  then  entered  into  the 
SWORD  computer  program  that  calculated  workload  and  situational  awareness 
ratings. 

4.6  Experimental  Design 


The  study  used  a  repeated  measures  2  x  2  x  2  x  3  full  factorial  experimental 
design.  The  design  consisted  of  2  levels  of  Decision  Aid  (aid,  no  aid),  2  levels  of  Task 
Difficulty  (low  task  load,  high  task  load),  2  levels  of  EAP  Orientation  (north-up,  track- 
up)  and  3  levels  of  Approach  Phase  (holding,  initial  approach,  final  approach). 


Each  testing  session  consisted  of  the  pilot  flying  three  approach  phases, 
consecutively  flown  in  accordance  with  the  instrument  approach,  with  one  of  the  eight 
testing  conditions  depicted  in  Table  4.  This  resulted  in  a  total  of  24  experimental 
conditions  (2  Decision  Aid  Conditions  x  2  Task  Difficulty  Levels,  2  EAP  Orientations 
X  3  Approach  Phases). 


Table  4.  Testing  Conditions  across  Approach  Phases 


DECISION  AID 

TASK  DIFFICULTY 

ORIENTATION 

AID 

HIGH  TASK  LOAD 

TRACK-UP 

NOAID 

HIGH  TASK  LOAD 

NORTH-UP 

AID 

LOW  TASK  LOAD 

TRACK-UP 

NOAID 

LOW  TASK  LOAD 

NORTH-UP 

AID 

HIGH  TASK  LOAD 

NORTH-UP 

NOAID 

HIGH  TASK  LOAD 

TRACK-UP 

AID 

LOW  TASK  LOAD 

NORTH-UP 

NOAID 

LOW  TASK  LOAD 

TRACK-UP 
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The  presentation  order  of  the  Decision  Aid  conditions  was  counterbalanced  across 
pilots  to  reduce  potential  order  effects.  The  presentation  orders  of  the  Task  Difficulty 
and  EAP  Orientation  conditions  were  blocked  with  respect  to  the  counterbalanced 
Decision  Aid  conditions. 
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5.  RESULTS  AND  DISCUSSION 


This  section  provides  a  summary  of  the  data  analysis  methods,  significant 
Analysis  of  Variance  (ANOVA)  results,  findings  from  the  questionnaire  data,  and  a 
discussion  of  the  combined  results  for  each  study  objective. 

5.1  Effectiveness  of  APES  (Objective  1) 

Objective  1  of  the  study  was  to  assess  the  effectiveness  of  APES  for  supporting 
approach  tasks  and  its  potential  for  reducing  pilot  workload,  increasing  situational 
awareness,  and  improving  performance.  The  following  sections  discuss  the  analysis 
and  study  findings  relevant  to  each  of  these  potential  benefits. 

Statistical  comparisons  of  RMS  Performance,  SWORD  Situational  Awareness 
and  Workload  ratings,  using  ANOVA  techniques,  were  performed  across  the 
experimental  conditions.  For  the  questionnaire  ratings,  frequencies  and  averages  were 
calculated  and  pilot  comments  were  summarized.  Appendix  B  provides  the  F- 
statistics,  significance  levels,  and  means  associated  with  the  tested  effects  of  the 
performance,  workload,  and  situational  awareness  data.  Appendix  C  provides  the 
questionnaire  ratings  and  summarized  pilot  comments. 

5.1.1  Effectiveness  of  APES  for  Supporting  Approach  Tasks 

Average  pilot  ratings  showed  that  APES  “moderately  enhanced”  task 
accomplishment  for  most  approach  tasks.  As  shown  in  Figure  4,  APES  benefited 
pilots  most  in  identifying  and  navigating  to  fixes. 


19 


Identify  Approach  Radials  /  Arc 
Determine  Turn  into  holding 
Develop  a  Mental  Picture 
Identify/ZNavigate  to  lAF 
Identity/Navigate  to  Holding  Fix 
Identify  Approach  Altitude 

1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.i 

SUBSTANTIALLY  NEUTRAL  SI 

HINDERED  £ 

ES  Influence  on  Task  Perl 

’  overall  utility  as  “extreme 


Figure  4.  Pilot  Ratings  of  API 

Two-thirds  of  the  pilots  rated  APES’ 


factors.  Figure  5  illustrates  this  interaction.  No  significant  results  were  found  between 
EAP  North-Up  and  Track-Up  Orientation. 
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Figure  5.  Decision  Aiding  and  Task  Difficulty  as  a  Function  of  Subjective 

Workload 

Questionnaire  results  substantiate  that  the  high  task  loading  condition  induced 
workload,  which  was  the  intent  of  the  experimental  design.  The  majority  of  pilots 
(60%)  thought  that  wind  gusts  and  no  prior  EAP  review  “moderately  increased 
workload.”  The  other  40%  thought  that  workload  was  “substantially  increased”  in  the 
high  task  loading  condition.  These  findings  are  consistent  with  the  SWORD  results 
which  showed  significantly  higher  workload  ratings  in  the  high  task  loading  condition. 

5.1.3  Situational  Awareness 

Situational  awareness  (SA)  was  defined  as  “the  ability  to  assess  current  position 
relative  to  the  approach  and  predict  future  actions.”  As  with  the  SWORD  workload 
ratings,  statistical  comparisons  of  the  SWORD  SA  levels,  using  ANOVA  techniques, 
were  performed  across  the  different  decision  aiding,  task  loading  and  EAP  Orientation 
conditions. 

As  shown  in  Figure  6,  pilots  rated  situational  awareness  as  being  significantly 
better  (p  <  0.01)  with  APES  than  without  APES.  This  effect  was  across  both  low  and 
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high  task  loading  conditions  and  EAP  North-Up  and  Track-Up  orientations.  A 
significant  effect  was  also  found  for  task  loading  {p  <  0.01),  indicating  that  pilots 
considered  their  SA  to  be  significantly  better  in  the  low  task  loading  condition  than  in 
the  high  task  loading  condition. 


LESS  SA 


AID 


NO  AID 


Figure  6.  Decision  Aiding  as  a  Function  of  Situational  Awareness 

The  questionnaire  responses  supported  the  SWORD  SA  ratings.  Two-thirds  of  the 
pilots  reported  that  their  situational  awareness  was  “substantially  enhanced”  with  the 
APES  advice. 

5.1.4  Pilot  Performance 

The  analysis  for  pilot  performance  was  conducted  by  assessing  RMS  altitude, 
airspeed,  heading,  course  and  glideslope  (precision  approach)  deviation  data  that  were 
collected  during  the  simulation  testing  sessions.  Statistical  comparisons  of  the  RMS 
data,  using  ANOVA  techniques,  were  performed  across  the  different  decision  aiding, 
task  loading  and  EAP  Orientation  and  Approach  Phase  conditions.  Unlike  the 
SWORD  Workload  and  Situational  Awareness  analysis.  Approach  Phase  (i.e.  Hold, 
Initial  Approach  and  Final  Approach),  was  included  as  a  factor  in  the  statistical 
analysis  of  the  RMS  performance  data.  One-way  ANOVAs  were  performed  on  any 
interaction  effects  to  identify  the  nature  of  the  statistical  differences. 
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The  performance  results  generally  showed  that  pilots  performed  better  with 
decision  aiding  than  without  decision  aiding.  Pilots  deviated  significantly  less  from 
assigned  altitude  with  APES  advice;  and  that  this  effect  was  most  prominent  in  the 
high  task  loading  condition.  Figure  7  illustrates  this  significant  interaction  {p  <  0.01). 
The  RMS  altitude  deviation  data  did  not  show  any  significant  effects  or  interactions 
for  Approach  Phase  or  EAP  orientation. 


□  AID  ■  NO  AID 


Figure  7.  Decision  Aiding  and  Task  Difficulty  as  a  Function  of  RMS 

Altitude  Deviation 

The  benefits  of  APES  may  also  be  influenced  by  approach  phase.  With  APES 
advice,  pilots  deviated  significantly  less  from  the  assigned  airspeed  during  the  initial 
approach  and  final  approach  phases,  than  during  holding.  Figure  8  illustrates  this 
significant  interaction  (p  <  0.01).  The  RMS  airspeed  deviation  data  did  not  show  any 
significant  effects  or  interactions  for  Task  Loading  or  EAP  Orientation. 


Figure  8.  Decision  Aiding  and  Approach  Phase  as  a  Function  of 

RMS  Airspeed  Deviation 


The  performance  results  did  not  show  any  decision  aid  effects  or  interactions  in 
the  heading,  course,  and  glideslope  RMS  deviation  data.  However,  it  was  found  that 
RMS  heading  and  course  deviations  were  significantly  smaller  in  the  Final  Approach 
Phase  than  during  the  Initial  Approach  Phase.  These  effects  are  not  surprising  given 
that  the  navigation  aid  for  the  final  approach  gives  more  accurate  and  sensitive  course 
information  than  the  navigation  aid  for  the  initial  approach. 

5.1.5  Effectiveness  of  APES  (Objective  1)  Discussion 

The  combined  results  (from  the  questionnaire,  workload,  situational  awareness, 
and  performance  data)  demonstrate  that  APES  was  effective  in  supporting  task 
performance;  and  in  reducing  workload,  increasing  situational  awareness,  and 
improving  performance.  The  altitude  deviation  and  workload  results  suggest  that 
APES  was  most  beneficial  during  high  task  loading  conditions.  The  airspeed  deviation 
results  indicate  that  APES  effectiveness  was  influenced  by  the  task  priority  and  the 
type  of  APES  advice  that  is  given  during  a  particular  approach  phase.  Because 
maintaining  airspeed  is  considered  to  be  a  more  critical  task  during  initial  and  final 
approach  phases  than  during  holding,  it  is  not  surprising  that  APES  airspeed  advice 
was  found  to  be  more  beneficial  during  these  two  phases. 

The  benefits  that  were  found  with  the  APES  airspeed  and  altitude  advice, 
however,  may  have  obscured  potential  benefits  associated  with  the  APES  course 
advice  and  heading  advice.  Because  pilots  simultaneously  maintain  course,  heading, 
altitude,  and  airspeed,  it  is  possible  that  course  and  heading  were  maintained  at  the 
expense  of  airspeed  and  altitude.  This  would  explain  why  APES  advice  was  shown  to 
improve  altitude  and  airspeed  performance,  but  not  heading  and  course  performance. 
The  effects  of  multi-tasking  may  also  explain  the  lack  of  decision  aid  benefits  for 
glideslope  deviation.  However,  it  is  also  possible  that  APES  did  not  provide  any 
value-added  glideslope  information  because  of  the  excellent  “guidance”  information 
that  is  provided  by  the  pitch  steering  bar  on  the  Attitude  Director  Indicator  (ADI). 
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5.2  Decision  Aid  Performance  (Objective  2) 


Objective  2  of  the  study  was  to  assess  the  performance  of  the  decision  aid  to 
determine  if  APES  advice  was  accurate  and  timely  enough  to  assist  the  pilot  in  flying 
instrument  approaches.  To  assess  APES  accuracy,  pilot  ratings  of  APES  logic  and 
consistency  were  analyzed.  To  assess  whether  APES  advice  was  given  at  the  time 
when  it  was  most  needed,  pilot  ratings  of  APES  timeliness  were  analyzed.  Frequencies 
and  averages  were  computed  for  the  questionnaire  ratings  and  pilot  comments  were 
summarized. 


5.2.1  APES  Logic 


Figure  9  shows  pilot  ratings  of  the  APES  logic  for  the  procedural  and  correction 
advice  (of  the  respective  “advisory  pilot”  and  “advisory  copilot”  components).  Logic 
was  defined  as  the  APES  advice  being  correct  and  complete.  For  all  advice  prompts, 
average  pilot  ratings  for  advice  logic  were  above  “moderately  acceptable”  (i.e,  above 
4.0). 
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Figure  9.  Pilot  Ratings  of  APES  Logic 


The  lower  ratings  of  the  airspeed  and  altitude  correction  advice  were  primarily  due 
to  the  deviation  tolerance  windows  that  were  established  to  activate  the  decision  aid. 
In  general,  pilots  indicated  that  the  +/-  100  feet  altitude  window  and  the  +/-  10  knots 


25 


airspeed  window  were  too  stringent,  especially  during  turbulent  conditions.  Several 
pilots  commented  that  they  would  like  to  have  the  option  of  setting  the  window  criteria 
themselves.  One  pilot  thought  that  the  APES  should  automatically  adjust  tolerance 
windows  based  on  current  or  future  operating  conditions. 

One  pilot  expressed  concern  regarding  the  potential  of  APES  giving  wrong 
information,  such  as  when  the  ATC  assigns  an  altitude  restriction  (which  would  be 
unknown  to  APES).  Five  pilots  reported  that  APES’  best  feature  was  its  capability  for 
automatic  setting  of  navaid  channels  /  frequencies. 

5.2.2  APES  Consistency 

Figure  10  shows  pilot  ratings  of  APES  procedural  and  correction  advice 
consistency.  Consistency  was  defined  as  the  APES  advice  being  consistently  given 
whenever  it  was  needed. 

For  all  advice  types,  the  average  ratings  for  advice  consistency  were  well  above 
“moderately  acceptable  (4.0).”  There  were  no  pilot  comments  regarding  the 
consistency  of  the  APES  advice. 
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Figure  10.  Pilot  Ratings  of  APES  Consistency 
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5.2.3  APES  Timeliness 


Figure  11  shows  pilot  ratings  of  the  APES  timeliness  for  the  procedural  and 
correction  advice.  Timeliness  was  defined  as  the  APES  advice  occurring  at  the  time 
that  it  is  needed.  For  all  advice  types,  average  pilot  ratings  for  advice  timeliness  were 
above  “moderately  acceptable.”  Course  and  radio  prompts  received  the  highest 
ratings. 

_n„p,p„iv  ■  PROCEDURAL  ADVICE  □  CORRECTION  ADVICE 


Figure  11.  Pilot  Ratings  of  APES  Timeliness 

Several  pilots  (3)  commented  that  missed  approach  procedural  prompts  were 
“late”  and  “hurried.”  Four  pilots  reported  that  APES  messages  came  too  close 
together  at  the  fixes  (i.e.,  lAF,  FAF)  and  that  they  should  be  spaced  farther  apart. 

5.2.4  APES  Performance  (Objective  2)  Discussion 

The  questionnaire  data  showed  that  pilots  considered  the  performance  of  the 
APES  to  be  acceptable  in  providing  both  “correction”  and  “procedural”  advice  to  the 
pilot.  Pilot  comments  regarding  the  APES  performance  primarily  concerned  the  logic 
and  timeliness  of  the  airspeed  and  altitude  corrrection  advice  and  the  stringent  altitude 
and  airspeed  windows  that  were  set  for  decision  aid  activation. 

Pilot  comments  that  “too  many  prompts  occurred  too  close  together”  at  the  fixes 
(i.e.,  lAF,  FAF)  is  contingent  upon  fixed  approach  requirements  which  can  not  be 
changed.  One  possible  way  of  minimizing  the  “hurried  appearance”  of  the  APES 
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voice  messages  would  be  to  present  the  lower  priority  messages  (or  a  specific  type  of 
message)  in  a  textual  format. 

5.3  Pilot-Vehicle  Interface  Understandability  and  Useability  (Objective  3) 

Objective  3  of  the  study  was  to  assess  the  understandability  and  useability  of  the 
pilot- vehicle  interface  to  determine  if  the  interface  allowed  the  pilot  to  easily  interpret 
and  use  APES  advice.  The  analysis  for  this  objective  focused  on  questionnaire 
responses.  Frequencies  and  averages  were  computed  for  the  questionnaire  ratings  and 
pilot  comments  were  summarized. 

5.3.1  Questionnaire  Results 

On  average,  pilots  rated  the  voice  messages  for  altitude,  airspeed,  heading,  course 
and  radio  advice  as  above  “moderately  acceptable.”  Most  pilots  (12)  indicated  that 
they  did  not  use  the  APES  message  display  on  the  right  CRT  because  it  was  located 
outside  their  cross-check. 


The  average  pilot  ratings  for  the  APES  verbal  advice  are  presented  in  Figure  12. 
As  shown,  pilots  average  ratings  were  above  “moderately  acceptable”  for  the 
parameters  of:  voice  quality,  sequence  of  voice  messages  and  understandability  of  the 
voice  messages.  The  average  rating  for  the  phraseology  of  the  voice  messages, 
however,  were  slightly  below  “moderately  acceptable.” 


Completely 

Acceptable 


Sequence 


Figure  12.  Pilot  Ratings  on  APES  Verbal  Advice 
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Several  pilots  (6)  commented  that  the  phraseology  of  the  airspeed  and  altitude 
verbal  procedural  advice  needed  to  be  more  specific.  Three  pilots  commented  that 

when  descending  to  decision  height,  the  decision  aid  should  say  “descend  to _ 

instead  of  “maintain _ Also,  two  pilots  commented  that  pauses  between 

verbal  messages  should  be  longer. 

Another  pilot  thought  that  the  voice  messages  may  clutter  the  radio  and  interfere 
with  ATC  communications  during  critical  phases  of  the  flight.  Two  pilots  indicated 
that  they  would  like  the  option  of  being  able  to  turn  the  APES  verbal  advice  “off.” 

Also,  one  pilot  commented  that  the  PVI  implementation  of  the  scratchpad  advice 
did  not  allow  for  meaningful  interpretation.  The  “Enter”  key  (to  set  appropriate 
command  markers  and  to  clear  the  scratchpad)  was  often  pressed  without  giving 
thought  of  the  value  being  manipulated. 

5.3.2  Pilot-Vehicle  Interface  Understandability  and  Useability  (Objective  3) 
Discussion 

The  results  showed  that  pilots  rated  the  interface  as  being  “acceptable,”  however, 
comments  indicated  that  the  phraseology  of  the  voice  advice  could  be  improved. 

APES  textual  display  of  advice  messages  also  warrants  further  examination.  Pilot 
comments  overwhelmingly  indicated  that  APES  textual  messages,  displayed  on  the 
right  CRT,  were  located  too  far  outside  the  pilot’s  cross-check  to  be  useful.  In  the 
current  implementation,  the  right  CRT  primarily  displayed  information  that  was 
redimdant  of  the  APES  voice  message  presentation.  Future  implementations  would 
need  to  consider  a  location  and  format  that  is  within  pilot’s  cross-check,  especially  if  it 
is  a  primary  APES  message  display. 
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6.  CONCLUSIONS  AND  RECOMMENDATIONS 


Overall,  the  results  provided  strong  evidence  indicating  that  the  prototype 
Approach  Procedure  Expert  System  (APES)  reduces  pilot  workload  and  improves 
flight  performance  compared  to  flying  approaches  without  the  decision  aid.  APES 
appears  to  be  most  beneficial  during  high  task-loading  conditions  where  the  pilot 
experienced  high  workload.  Also,  the  majority  of  the  pilots  commented  that  that  their 
situational  awareness  was  “substantially  enhanced”  with  the  APES  advice  and 
significant  SA  ratings  support  this  finding.  Also,  all  pilots  commented  that  they  felt 
that  APES  would  improve  flight  safety. 

Although  refinements  were  indicated,  the  results  also  showed  that  the  APES 
“advisory  pilot”  and  “advisory  copilof  ’  components  consistently  gave  advice  whenever 
it  was  needed,  and  that  advice,  for  the  most  part,  was  complete  and  accurate  for 
performing  all  tasks  in  all  phases  of  flight.  Finally,  the  results  indicated  that  the  pilot- 
vehicle  interface  was  presented  in  a  format  that  was  both  intelligible  and  operable  for 
APES  advice  delivery,  although  refinements  were  again  indicated. 

6.1  Expert  System  General  Design  Considerations 

As  with  any  decision  aid,  careful  consideration  needs  to  be  given  to  the  known 
disadvantages  associated  with  semi-automated  systems.  Because  low-level  decision 
making  processes  are  automated,  the  pilot  may  view  the  system  as  a  black  box  that 
generates  outputs  from  inputs  through  some  unknown  mechanism,  such  as  the 
algorithm.  This  may  impair  pilot  confidence  in  the  system  and  result  in  the  pilot 
completely  ignoring  the  advice  of  the  decision  aid.  Conversely,  too  much  trust  and 
over-reliance  on  the  decision  aid  may  lead  to  reduced  pilot  situational  awareness, 
which  in  turn,  could  adversely  affect  flight  performance  if  the  decision  aiding  system 
fails  or  an  emergency  occurs. 

One  way  to  mitigate  the  possible  effects  of  reduced  situational  awareness  and 
system  confidence  is  through  proper  training  of  the  decision  aid  logic.  This  training 
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would  enable  the  pilot  to  develop  an  accurate  mental  model  of  the  reasoning  behind 
the  advice  (Lehner  and  Zirk,  1987).  Equally  important  is  proper  design  of  the  pilot- 
vehicle  interface  to  facilitate  the  human-computer  interaction  and  allow  the  pilot  to 
easily  interpret  the  decision  aid  advice. 

An  effective  decision  aid  may  also  need  to  include  user-selectable  options  as  part 
of  its  design,  giving  the  pilot  flexibility  in  configuring  the  PVI.  For  example,  the  pilot 
may  find  it  useful  to  configure  display  modes  (audio  or  visual)  for  certain  types  of 
advice  (e.g.,  altitude,  airspeed,  course).  The  pilot  may  also  find  it  beneficial  to  set  the 
priority  levels  (e.g.,  primary  and  secondary)  of  the  various  advice  types,  as  well  as, 
adjust  their  tolerance  windows  (e.g.,  +/-  100  feet  for  altitude  deviation). 

6.2  Implications  for  Future  Research 

The  results  of  this  study  strongly  encourage  future  development  of  the  approach 
plate  expert  system.  We  believe  that  the  use  of  the  APES  could  ultimately  reduce  the 
number  of  incidents  and  accidents  during  approach. 

Enhancement  of  the  APES  algorithm  is  one  area  for  follow-on  research.  Future 
implementations  of  APES  altitude  and  airspeed  correction  advice  may  need  to  consider 
trend  information.  In  its  current  implementation,  APES  could  not  provide  explicit 
corrective  altitude  and  airspeed  advice  (i.e.,  increase/decrease  by  XXX  feet/XX  knots) 
because  altitude  and  airspeed  were  too  dynamic  for  accurate  input.  APES  procedural 
advice  could  also  be  augmented  to  provide  a  predictive  rate  of  descent  (i.e.,  VVI)  to 
capture  target  altitude.  In  its  current  implementation,  APES  only  periodically 
prompted  the  pilot  of  the  target  altitude. 

The  APES  PVI  is  another  area  for  follow-on  research.  Future  APES 
implementations  will  need  to  address  potential  conflicts  between  the  APES  audio 
advice  and  radio  communications.  PVI  developmental  efforts  should  also  focus  on 
optimum  formats  for  critical  phases  of  flight  where  simultaneous  changes  on  multiple 
flight  parameters  are  required  (e.g.,  lAF,  FAF,  MAP).  This  research  should  investigate 
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pilot  selectable  options  for  configuring  the  PVI  (e.g.,  setting  deviation  tolerance 
windows  for  decision  aid  activation).  Issues  involved  in  implementing  an  intelligent 
agent  for  automatic  adjustments  of  display  and  advice  settings  should  also  be 
examined. 

Future  pilot-in-the-loop  studies  should  evaluate  APES  to  assess  its  robustness 
across  a  diversity  of  approach  types,  including  high-altitude  approaches  (only  low- 
altitude  were  used  in  the  current  study)  and  non-typical  approaches.  These  evaluations 
should  include  air  traffic  control  communications  as  part  of  the  testing  scenario. 
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APPENDIX  A  -  QUESTIONNAIRE  AND  SUBJECTIVE 
WORKLOAD  DOMINANCE  (SWORD)  FORMS 
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Introduction 


The  questionnaires  and  SWORD  forms  that  were  used  in  the  Pilot-in-the-Loop  Evaluation  of  the 
Approach  Procedures  Expert  System  (APES)  are  provided  in  this  Appendix.  The  questionnaire 
responses  of  the  16  pilots  who  participated  in  this  study  are  summarized  in  Appendix  C.  A 
summary  of  the  SWORD  responses  is  presented  in  Appendix  B. 
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SESSION  QUESTIONNAIRE 

Subject  No.  _ 

Session  No.  _ 

Decision  Aid:  Yes  /  No  Orientation:  N-Up  /  T-Up  Prior  Review:  Yes  /  No 

1 .  Use  the  scale  below  to  rate  the  ease  of  performing  the  following  tasks  for  this  approach.  For  response 
ratings  of  “D”  or  “E,”  please  explain  under  comments. 

{Rating  Scale: 

A  ==  Extremely  Easy 
B  ==  i^oderately 
C  -  Neutral 
D  =  Moderately  Difficult 
E  =  Extremely  Difficult 

Comments: 

a)  _  Assessing  aircraft  performance 

b)  _  Flying  the  holding  pattern 

c)  _  Flying  the  Initial  Approach  track 

d)  _  Flying  the  Final  Approach 

e)  _  Executing  Missed  Approach  procedures 

f)  _  Overall  Rating  of  Approach  Tasks 

2.  What  are  your  initial  comments  regarding  this  approach  and  (if  applicable)  the  decision  aid? 
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FINAL  QUESTIONNAIRE 


Subject  No. 


Instructions:  Use  the  scale  below  to  rate  the  quality  of  each  type  of  “PROCEDURAL”  advice  given 
in  each  segment  of  the  approach. 


L  PROCEDURAL  ADVICE 


;;A  “  Completely  Acceptable: 

No  changes  needed;  acceptable  a$  is. 

B  -  Moderately  Acceptable: 

V  Minor  changes  desirable;  effectiveness  not  impaired* 
C  ==  Borderline: 

Changes  desirable;  effectiveness  may  be  impaired. 

D  ~  Moderately  Unacceptable; 

;  Changes  required,  effectiveness  moderately  impaired. 

E  =  Completely  Unacceptable; 

_ Redesign  required,  effectiveness  severely  impaired.' 


HEADING  PROMPTS 


HOLDING 

INITIAL  APPROACH  TRACK 

FINAL  APPROACH  TRACK 

MISSED  APPROACH 

OVERALL 

Comments: 


Comments: 


L  PROCEDURAL  ADVICE  (CONT,) 


Subject  No. 


Instructions:  Use  the  scale  below  to  rate  the  quality  of  each  type  of  “PROCEDURAL”  advice  given 
in  each  segment  of  the  approach. 


A  =  Completely  Acccpteble: 

No  ehanges;,nee^ed^  acceptable  as  is. 

B  -  Moderately;Aeceptab1e: 

Minor  desirable;  effectiveness  not  impaired. 

C  -  Borderlinef^T^ 

Cliangea>4esirable;  effectiveness  may  be  impaired. 

D  -  Moderately  Unacceptabicj 

Changes  required^^sieffectiveness  moderately  impaired 
E  -  Completely  Unacceptable: 

_ R^design.reqnircd,  effectiveness  severely  impaired. 


AIRSPEED  PROMPTS 


HOLDING 

INITIAL  APPROACH  TRACK 

FINAL  APPROACH  TRACK 

MISSED  APPROACH 

OVERALL 

Camments: 
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I.  PROCEDURAL  ADVICE  (CONT.) 


Subject  No. 

Instructions:  Use  the  scale  below  to  rate  the  quality  of  each  type  of  “PROCEDURAL”  advice  given 
in  each  segment  of  the  approach. 


A  =  Completely  Acceptable: 

No  changes  needed;  acceptable  as  is. 

-  Moderately  Acceptable: 

Minor  changes  desirable;  effectiveness  not  Impaired. 
C  ~  Bordelrlihe:  V 

Chktiges  desirable;  effectiveness  may  be  impaired. 

D  =  Moderately  Unacceptable: 

.  Changes  required,  effectiveness  moderately  impaired. 
E  -  Completely  Unacceptable: 

_ Redesign  required,  effectiveness  severely  impaired. 


RADIO  FREQ/CHANNEL  PROMPTS 


HOLDING 

INITIAL  APPROACH  TRACK 

FINAL  APPROACH  TRACK 

MISSED  APPROACH 

OVERALL 

Comments: 


11.  DEVIATION  ADVICE 

Subject  No,  _ 

Instructions:  Use  the  scale  below  to  rate  the  quality  of  each  type  of  “DEVIATION”  advice  given  during  all 
segments  of  the  approach. 


Comments: 


A-7 


III.  TASK  ACCOMPLISHMENT  AND  EAP  ORIENTATION 


Subject  No. 


?  Rating  Scale; 

A  =  Substantially  Enhanced 
B  =  Moderately  Enhanced 
C  -  Did  Not  Affect 
D  -  Moderately  Degraded 
\  E  ==  Substantially  Degraded 


Instructions:  Use  the  scale  above  to  rate  the  effectiveness  of  the  decision  aid  versus  no  decision  aid  for 
performing  the  following  tasks  in  Track-Up  EAP  Orientation,  North-Up  EAP  Orientation,  and  Overall. 
Please  provide  comments  for  “D”  or  “E”  ratings. 


The  Dec^ibn  Aid  my  ability  to  . . . 

Track- 

Up  , 

North- 

Up 

Overall 

Comments; 

a)  identify  and  navigate  to  the  holding 
fix. 

b)  determine  direction  of  turn  into 
holding. 

c)  identify  and  navigate  to  the  lAF. 

d)  identify  approach  radials  /  arc. 

e)  identify  approach  altitude. 

f)  identify  and  navigate  to  the  FAF. 

g)  identify  the  decision  height  or 
minimum  descent  altitude. 

h)  identify  and  navigate  to  the  MAP. 

i)  identify  and  set  Navaids. 

j)  determine  intercept  headings  /  angles. 

k)  determine  lead  points  for  arcs  / 
courses. 

1)  determine  airspeeds. 

m)  determine  aircraft’s  position  relative 
to  approach  fixes. 

n)  develop  a  mental  picture  for 
anticipating  future  events. 

IV.  PILOT- VEHICLE  INTERFACE 
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Subject  No:  _ 

Rating  Scale: 

A  =  Completely  Acceptable:  No  changes  needed;  acceptable  as  is. 

B  =  Moderately  Acceptable:  Minor  changes  desirable;  effectiveness  not  impaired. 

C  =  Borderline:  Changes  desirable;  effectiveness  may  be  impaired. 

D  =  Moderately  Unacceptable:  Changes  required,  effectiveness  moderately  impaired. 

E  =  Completely  Unacceptable:  Redesign  required,  effectiveness  severely  impaired. 

Instructions:  Use  the  above  scale  to  rate  the  overall  quality  of  the  pilot-vehicle  interface"'  for  the  following 
prompts.  For  response  ratings  of  “C”  or  below,  please  identify  the  parameter(s)  that  is  deficient  by  placing 
a  check  (V)  on  the  appropriate  line.  Also  please  explain  under  comments. 


Comments: 

1 .  _  Heading  Prompts  (overall) 

a)  _ Audio  Display  (Voice) 

b)  _  Visual  Display  (CRT) 

c)  _  Frequency  of  Prompt  Repetition 

d)  _  Tolerance  Window  (+/-  5° ) 

2.  _  Altitude  Prompts  (overall) 

a)  _  Audio  Display  (Voice) 

b)  _  Visual  Display  (CRT) 

c)  _  Frequency  of  Prompt  Repetition 

d)  _ Tolerance  Window  (+/- 100  ft) 

3.  _  Airspeed  Prompts  (overall) 

a)  _  Audio  Display  (Voice) 

b)  _  Visual  Display  (CRT) 

c)  _  Frequency  of  Prompt  Repetition 

d)  _ Tolerance  Window  (+/-  10  kts) 

4.  _  Course  Prompts  (overall) 

a)  _  Audio  Display  (Voice) 

b)  _  Visual  Display  (CRT) 

c)  _  Frequency  of  Prompt  Repetition 

d)  _  Tolerance  Window  (+/-2‘^ ) 

5.  _ Frequency/Channel  Prompts  (overall) 

a)  _  Audio  Display  (Voice) 

b)  _  Visual  Display  (CRT) 

c)  _  Frequency  of  Prompt  Repetition 


Additional  Comments: 
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IV.  PILOT- VEHICLE  INTERFACE  (continued) 


Subject  No. 


Rating  Scale: 

A  =  Completely  Acceptable:  No  changes  needed;  acceptable  as  is. 

B  =  Moderately  Acceptable:  Minor  changes  desirable;  effectiveness  not  impaired. 

C  =  Borderline:  Changes  desirable;  effectiveness  may  be  impaired. 

D  =  Moderately  Unacceptable:  Changes  required,  effectiveness  moderately  impaired. 

E  =  Completely  Unacceptable:  Redesign  required,  effectiveness  severely  impaired. 

Instructions:  Use  the  scale  above  to  rate  the  effectiveness  of  Pilot- Vehicle  Interface.  For  response  ratings 
of  “C”  or  below,  please  explain  under  comments. 


Comments: 

6.  Audio  (Voice)  parameters: 

a)  _  Voice  Quality /Ease  of  Hearing  Voice  Messages 

b)  _  Understandability  /  Comprehensiveness  of  Voice  Messages 

c)  _ Phraseology  of  Voice  Message 

d)  _  Sequence  of  Voice  Message  Format 

7.  In  what  ways  could  the  Voice  Interface  be  improved? 


8.  Visual  (CRT)  parameters: 

a)  _  Readability  of  the  Messages 

b)  _ Understandability  /  Comprehensiveness  of  Text  Messages 

c)  _  Phraseology  of  Text  Messages 

d)  _  Sequence  of  Message  Presentations 

e)  _  Location  of  Text  Messages 

9.  In  what  ways  could  the  Text  Message  Interface  be  improved? 
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V.  GENERAL  QUESTIONS 


Subject  No. 

1 .  How  frequently  did  you  follow  the  procedural  advice  of  the  decision  aid? 

_  Always 

_  Often 

_  Sometimes 

_ Almost  Never 

_ Never 

Comments: 

2.  When  given,  how  ifequently  did  you  follow  the  deviation  advice  of  the  decision  aid? 
_  Always 

_  Often 

_  Sometimes 

_  Almost  Never 

_  Never 

Comments: 

3.  Did  EAP  orientation  (North-Up/Track-Up)  affect  your  use  of  the  decision  aid? 

_ YES  _ NO 

If  YES y  in  what  ways? 

4.  The  combination  of  wind  gusts  and  no  prior  review  of  the  approach  plates: 

_  Did  not  affect  workload. 

_  Moderately  increased  workload. 

_  Substantially  increased  workload. 

Comments: 


5.  How  did  wind  gusts  and  no  prior  review  of  the  approach  plate  affect  your  use  of  the  decision  aid? 

_  Substantially  increased  usage 

_ Moderately  increased  usage 

_  Did  not  affect  usage 

_  Moderately  decreased  usage 

_  Substantially  decreased  usage 

Comments: 
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V.  GENERAL  QUESTIONS  (cont.) 


6.  The  decision  aid  improves  safety  when  flying  instrument  approaches. 

_  Strongly  agree 

_  Moderately  agree 

_  Neutral 

_  Moderately  disagree 

_  Strongly  disagree 

Comments: 

7.  How  would  you  rate  the  overall  utility  of  the  decision  aid  for  assisting  you  in  performing  instrument 
approaches. 

_  Extremely  Beneficial 

_  Moderately  Beneficial 

_  Neutral 

_  Moderate  Hindrance 

_  Extreme  Hindrance 

Comments: 


8.  Are  there  other  conditions,  not  simulated  here,  where  you  think  this  decision  aid  might  be  useful? 

_ YES  _ NO  _ NOT  SURE 

Comments: 

9.  Did  you  develop  a  particular  strategy  for  using  the  decision  aid? 

_ YES  _ NO 

If  YES,  please  comment. 


10.  What  did  you  like  best  about  the  decision  aid? 


1 1 .  What  did  you  like  least  about  the  decision  aid? 


12.  Additional  Comments: 
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SWORD  QUESTIONNAIRE  INSTRUCTIONS 


The  SWORD  technique  assesses  workload  /  situation  awareness  by  utilizing  a  series  of  pairwise 
comparisons  between  system  configurations.  For  this  study,  you  will  be  asked  to  make  comparisons 
between:  2  decision  aid  conditions  (aid  and  no  aid);  2  winds/approach  review  conditions  (with  winds,  no 
review  and  no  winds,  review),  and  2  EAP  orientation  conditions  (track-up  and  north-up). 

Rate  to  the  best  of  your  ability,  the  display  configuration  that  you  think  causes  higher  workload  or  promotes 
better  situational  awareness  (given  questionnaire  type).  Base  your  responses  on  your  experience  flying 
approaches  in  the  MAGIC  simulator. 

The  examples  below  show  comparison  ratings  that  can  be  made  between  various  conditions  of  this  study. 


Example  1-  shows  that  the  AID  TRACK-UP  causes  substantially  more  workload  or  SA  {depending  on  scale) 
than  the  AID  NORTH-UP  when  winds  are  present  and  with  no  approach  review  (WNDS/NORVW). 


EQUAL 

AID  WNDS/NORVW  TRK-UP _ V _ | _ |_ 

NORTH-UP 


«  «<  «« 

AID  WNDS/NORVW 


Example  2  -shows  that  the  AID  NORTH  -UP  causes  moderately  more  workload  or  SA  (depending  on  scale) 
than  the  AID  TRACK-UP  when  winds  are  present  and  with  no  approach  review  (WNDS/NORVW). 


»>  >»  »  >  <  «  «<  «« 

EQUAL 

AID  WNDS/NORVW  TRK-UP _ _ _ _ | _ | _ V _ AID  WNDS/NORVW 

NORTH-UP 


Example  3  -  shows  that  workload  or  SA  (depending  on  scale)  between  the  AID  NORTH  -UP  condition  and 
the  NOAID  TRACK-UP  conditions  are  equal  when  winds  are  present  and  with  no  approach  review 
(WNDS/NORVW). 


»>  »>  »  > 

EQUAL 

AID  WNDS/NORVW  NORTH-UP _ l_V_|  _ 

TRK~UP 


<  «  «<  «« 

_  NOAID  WNDS/NORVW 
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WHICH  CONFIGURATION  ELICITS  THE  MOST  WORKLOAD? 
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DECISION  AID:  APPROACH  TYPE:  EAP  ORIENTATION: 

With  Aiding  (Aid)  No  Winds,  Review  of  EAP  (NoWnds/Rvw)  Trk-Up  (Track-Up) 
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WHICH  CONFIGURATION  ELICITS  THE  MOST  WORKLOAD? 


a:  0^  0^  cr 


^  ^  ^  <E 


DC  a:  cn  o:: 


■O  "O  "O 
<  <  < 


O) 


^  £- 
Q  TZJ 
W  < 
£ 

LU  ^ 
Q  5 


Without  Aiding  (NoAid)  Winds,  No  EAP  Review  (Wnds/NoRvw)  North-Up 


WHICH  CONFIGURATION  PROMOTES  BETTER  SITUATIONAL  AWARENESS? 
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APPENDIX  B  -  ANALYSIS  OF  VARIANCE  RESULTS 
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STATISTIC  ANALYSIS  SUMMARY 


1.  Introduction 

The  results  of  the  analysis  procedures  performed  on  the  performance,  situational  awareness, 
and  workload  data  are  provided  in  this  Appendix.  Each  data  group  will  provide  a  table  of  the 
results  of  the  ad-hoc  Analysis  of  Variance  (ANOVA)  procedures,  including  F  statistics, 
degrees  of  freedom,  and  significance  levels  for  each  main  effect  and  interaction.  Additional 
tables  will  show  the  means  used  in  the  ANOVA  analysis.  The  results  of  one-way  ANOVAs 
used  in  the  analysis  of  the  significant  interactions  are  also  described.  A  table  of  the 
applicable  means  used  in  each  interaction  is  presented. 

2.  Performance  Data 

The  analyses  treated  in  this  section  pertain  to  the  performance  data  collected  during  the 
instrument  approach  flying  sessions.  The  analyses  will  be  presented  separately  for  each  of 
the  following  dependent  performance  variables:  RMS  altitude  deviation,  RMS  airspeed 
deviation,  RMS  heading  deviation,  RMS  course  deviation,  and  RMS  glideslope  deviation. 

2.1  Root  Mean  Square  (RMS)  Altitude  Deviation 

A  2x2x2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  EAP  Orientation  conditions,  2  Approach  Phase  conditions)  was  performed  on 
the  Root  Mean  Square  (RMS)  altitude  deviation  data.  Because  RMS  Altitude  deviation  was 
not  appropriate  for  the  Final  Approach  phase  (the  aircraft  is  descending),  only  the  Holding 
and  Initial  Approach  phases  were  included  in  the  analysis.  The  results  are  provided  in  Table 
B-1 .  The  means  used  in  this  analysis  are  provided  in  Table  B-2. 
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Table  B-1.  ANOVA  Results  for  Altitude  Deviation 


Tested  Effect 

ANOVA  Results 

Decision  Aid 

F(l, 15)  =  26.77,  p  =  . 000* 

Task  Loading 

F(l,15)=  8.27,  p  =  . 009* 

EAP  Orientation 

F(l,15)=  0.08,  p  =  . 781 

Approach  Phase 

F(l,15)=  1.68,  p  =  . 215 

Decision  Aid  x  Task  Loading 

F(l,15)=  8.03,  p  =  . 013* 

Decision  Aid  x  EAP  Orientation 

il 

O 

b 

1! 

bo 

Decision  Aid  x  Approach  Phase 

F(l,15)=  6.40,  p  =  . 023* 

Task  Loading  x  EAP  Orientation 

F(l,15)=  0.12,  p  =  . 730 

Task  Loading  x  Approach  Phase 

F(l,15)=  2.07,  p  =  . 171 

Approach  Phase  x  EAP  Orientation 

F(l,15)=  0.19,  p  =  . 672 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F(l,15)=  0.00,  p  =  . 962 

Decision  Aid  x  Task  Loading  x  Approach  Phase 

F(l,15)=  0.45,  p  =  . 511 

Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(l,15)=  0.66,  p  =  . 429 

Decision  Aid  x  Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(l,15)=  0.31,  p  =  . 588 

*  indicates  statistical  significance,  /?  <  0  .05 


Table  B-2.  Means  Used  in  the  RMS  Altitude  Deviation  (feet)  Analysis 


HOLDING 

AID 

NOAID  1 

North-Up 

Track-Up 

Overall 

E!^559i 

Track-Up 

Overall 

51.44 

50.25 

58.79 

55.97 

57.38 

52.47 

55.38 

53.92 

68.10 

76.15 

72.13 

1  Overall 

51.96 

52.52 

52.08 

63.45 

66.06 

64.57 

INITIAL  APPROACH 

AID 

NOAID  1 

North-Up 

Track-Up 

Overall 

mmsm 

46.73 

44.54 

45.64 

67.51 

69.00 

52.68 

52.21 

■Us&H 

100.81 

93.00 

96,91 

1  Overall 

49.71 

48.38 

85.66 

73.65 

ALL  PHASES 

AID 

NOAID 

Overall 

North-Up 

Overall 

North-Up 

Overall 

49.08 

46.80 

47.94 

64.02 

61.74 

62.83 

55.39 

52.57 

53.80 

53.18 

83.93 

84.10 

68.64 

1  Overall 

50.83 

50.30 

50.56 

73.47 

62.02 
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Significant  main  effects  were  found  for  Decision  Aid  (F  =  16.11,  p  <  0.001)  and  Task 
Loading  (F  =  8.27,/?  =  0.009);  and  significant  interactions  were  found  between  Decision  Aid 
and  Task  Loading  conditions  (F  =  8.03,  p  =  0.013)  and  between  Decision  Aid  and  Approach 
Phase  conditions  (F  =  6.40,  p  =  0.023).  No  other  significant  main  effects  or  interactions 
were  found.  One-way  ANOVAs  were  performed  to  analyze  the  nature  of  the  significant 
interactions. 

For  the  Decision  Aid  by  Task  Loading  interaction,  a  simple  main  effect  was  found  for 
Task  Loading  in  the  No  Aid  condition  (F  =  1 1 .34,  p  =  0.004),  but  not  in  the  Aid  condition. 
In  the  No  Aid  condition,  RMS  altitude  deviation  was  significantly  smaller  with  Low  Task 
Loading  (mean  =  62.83  feet)  than  with  High  Task  Loading  (mean  =  84.10  feet).  The  means 
used  in  this  analysis  are  presented  in  Table  B-3. 

Table  B-3.  Means  Used  in  Analysis  of  Decision  Aid  by  Task  Loading  Interaction 


RMS  Altitude  Deviation  (feet) 

Aid 

NoAid 

Low  Task  Loading 

47.94 

62.83 

High  Task  Loading 

53.19 

84.10 

For  the  Decision  Aid  by  Approach  Phase  interaction,  no  significant  simple  main  effects 
were  found  for  Approach  Phase  for  either  the  Aid  or  No  Aid  conditions;  a  however,  a  trend 
(p  =  .056)  showed  smaller  RMS  altitude  deviations  were  found  in  the  Aid  condition  (mean  = 
59.04  feet)  compared  to  the  No  Aid  condition  (mean  =  82.95  feet)  during  the  Initial  Approach 
phase.  The  means  used  for  this  analysis  are  presented  in  Table  B-4. 


Table  B-4.  Means  Used  in  Analysis  of  Decision  Aid  by  Task  Loading  Interaction 


RMS  Altitude  Deviation  (feet) 

Aid 

NoAid 

Holding 

52.09 

64.57 

Initial  Approach 

59.04 

82.95 
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2.2  RMS  Airspeed  Deviation 


A  2x2x2x3  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  EAP  Orientation  conditions,  3  Approach  Phase  conditions)  was  performed  on 
the  Root  Mean  Square  (RMS)  airspeed  deviation  data.  The  results  are  provided  in  Table  B-5 
and  the  means  used  in  this  analysis  are  presented  in  Table  B-6. 

Table  B-5.  ANOVA  Results  for  RMS  Airspeed  Deviation 


Tested  Effect 

ANOVA  Results 

Decision  Aid 

F(l, 15)  =  11.15,  p  =  . 004* 

Task  Loading 

EAP  Orientation 

Approach  Phase 

F(2,30)  =  50.10,  p  =  . 000* 

Decision  Aid  x  Task  Loading 

F(l,15)=  0.20,  p  =  . 657 

Decision  Aid  x  EAP  Orientation 

F(l,15)=  0.99,  p  =  . 336 

Decision  Aid  x  Approach  Phase 

F(2,30)=  6.58,  p  =  . 004* 

Task  Loading  x  EAP  Orientation 

Lh 

's— ^ 

II 

p 

11 

a^ 

kO 

o 

Task  Loading  x  Approach  Phase 

F(2,30)=  14.55  p=. 000* 

Approach  Phase  x  EAP  Orientation 

F(l,15)=  0.10,  p  =  . 907 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F(l,15)=  1.67,  p  =  . 215 

Decision  Aid  x  Task  Loading  x  Approach  Phase 

F(2,30)=  1.78,  p  =  .  187 

Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(2,30)=  0.12,  p  =  . 885 

Decision  Aid  x  Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(2,30)=  1.92,  p  =  . 164 

*  indicates  statistical  significance,  p  <  0  .05 


Table  B-6.  Means  Used  for  RMS  Airspeed  Deviation  Analysis 


HOLDING 

AID 

NOAID  1 

North-Up 

Track-Up 

Overall 

North-Up 

Track-Up 

Overall  | 

12.89 

13.39 

wmam 

12.91 

12.19 

High  Task  Loading 

13.46 

11.75 

12.61 

11.73 

13.95 

Overall 

13.18 

12.57 

12.88 

12.32 

13.07 

12.70  1 

INITIAL  APPROACH 

AID 

1 

North-Up 

Track-Up 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

10.26 

11.43 

10.84 

15.28 

14.51 

14.90 

12.84 

10.92 

11.88 

14.30 

15,87 

15.08 

1  Overall 

11.55 

11.18 

11.36 

14.79 

15.19 

14.99 
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Table  B-6  (cont.)  Means  Used  for  RMS  Airspeed  Deviation  (knots)  Analysis 


FINAL  APPROACH 

AID 

North-Up 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

10.26 

11.43 

10.84 

15.28 

14.51 

14.90 

12.84 

11.88 

HBEQHI 

15.87 

mmm 

1  Overall 

11.55 

11.36 

15.19 

mmm 

ALL  PHASES 

AID 

NOAID 

Overall 

North-Up 

Track- 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

9.56 

10.44 

10.00 

11.43 

11.41 

11.42 

10.71 

11.78 

10.36 

11.07 

12.13 

13.74 

12.94 

12.01 

1  Overall 

10.67 

10.40 

10.54 

11.78 

12.58 

12.18 

11.36 

Significant  main  effects  were  found  for  Decision  Aid  (F  =  11.15,  p  =  0.004)  and 
Approach  Phase  (F  =  50.10,  p  <  0.001);  and  significant  interactions  were  found  between 
Deeision  Aid  and  Task  Loading  conditions  (F  =  6.58,  jt?  =  0.004)  and  between  Task  Loading 
and  Approach  Phase  conditions  (F  =  14.45,  p  <  0.000).  No  other  significant  main  effects  or 
interactions  were  found.  One-way  ANOVAs  were  performed  to  analyze  the  nature  of  the 
significant  interactions. 

For  the  Decision  Aid  by  Approach  Phase  interaetion,  a  simple  main  effect  was  found  for 
Decision  Aid  for  the  Initial  Approach  (F  =  1 1 .54,  p  =  .004)  and  the  Final  Approach  phases  (F 
=  14.27,  p  =  .002),  but  not  for  the  Holding  phase.  RMS  airspeed  deviation  was  significantly 
smaller  in  the  Aid  condition  (compared  to  No  Aid  condition)  for  both  the  Initial  Approach 
(Aid  mean  =  1 1.36  knots  /  No  Aid  mean  =  14.96  knots)  and  the  Final  Approaeh  phases  (Aid 
mean  =  7.57  knots  /  No  Aid  mean  =  9.08  knots).  The  means  used  in  this  analysis  are 
presented  in  Table  B-7. 
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Table  B-7.  Means  Used  in  Analysis  of  Decision  Aid  by  Approach  Phase  Interaction 


RMS  Airspeed  Deviation  (knots) 

Aid 

NoAid 

Holding 

12.88 

12.69 

Initial  Approach 

11.36 

14.96 

Final  Approach 

7.57 

9.08 

For  the  Task  Loading  by  Approach  Phase  interaction,  a  simple  main  effect  was  found  for 
Task  Loading  for  the  Final  Approach  phase  (F=23.43,  p  <  0.001),  but  not  for  the  Holding  and 
Initial  Approach  phases.  For  the  Final  Approach  phase,  RMS  airspeed  deviation  was 
significantly  smaller  in  the  Low  Task  Loading  (6.68  knots)  compared  to  High  Task  Loading 
condition  (9.97  knots).  Means  used  for  this  analyses  are  provided  in  Table  B-8. 


Table  B-8.  Means  Used  in  the  Analysis  of  Task  Loading  by  Approach  Phase  Interaction 


RMS  Airspeed  Deviation  (knots) 

Low  Task  Loading 

High  Task  Loading 

Holding 

12.65 

12.72 

Initial  Approach 

12.83 

13.33 

Final  Approach 

6.68 

9.97 

2.3  RMS  Heading  Deviation 

A  2x2x2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  LAP  Orientation  conditions,  2  Approach  Phase  conditions)  was  performed  on 
the  Root  Mean  Square  (RMS)heading  deviation  data.  Because  of  the  potential  for  spurious 
data  in  the  Holding  phase  (due  to  numerous  heading  changes),  only  Initial  and  Final 
Approach  phases  were  included  in  the  analysis.  The  results  of  the  ANOVA  analysis  are 
provided  in  Table  B-9  and  the  means  used  in  this  analysis  are  presented  in  Table  B-10. 

A  significant  main  effect  was  found  for  Approach  Phase  (F  =  113.49,  p  <  .000).  RMS 
heading  deviation  was  smaller  in  the  Final  Approach  phase  (mean  =  4.28  degrees)  than  in  the 
Initial  Approach  phase  (mean  =  12.21  degrees).  This  result  is  not  surprising  given  that  the 
navigation  aid  for  the  final  approach  gives  more  accurate  and  sensitive  course  information 
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than  the  navigation  aid  for  the  initial  approach.  No  other  significant  main  effects  or 
interactions  were  found. 


Table  B-9.  ANOVA  Results  for  RMS  Heading  Deviation 


Tested  Effect 

ANOVA  Results  | 

Decision  Aid 

Task  Loading 

EAP  Orientation 

F(l,15)  = 

Approach  Phase 

F(l,15)  = 

1 13.49,  p  =  . 000* 

Decision  Aid  x  Task  Loading 

F(l,15)  = 

0.23,  p  =  .639 

Decision  Aid  x  EAP  Orientation 

laUESB 

0.68,  p  =  .422 

Decision  Aid  x  Approach  Phase 

ISESB 

0.03,  p  =  .865 

Task  Loading  x  EAP  Orientation 

F(l,15)  = 

Task  Loading  x  Approach  Phase 

F(l,15)  = 

1.49,p  =  .241 

Approach  Phase  x  EAP  Orientation 

F(l,15)  = 

0.04,  p  =  .850 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F(l,15)  = 

0.75,  p  =  .399 

Decision  Aid  x  Task  Loading  x  Approach  Phase 

F(l,15)  = 

0.28,  p  =  . 601 

Task  Loading  x  EAP  Orientation  x  Approach  Phase 

0.39,  p  =  . 540 

Decision  Aid  x  Task  Loading  x  EAP  Orientation  x  Approach  Phase 

0.80,  p  =  . 384 

*  indicates  statistical  significance,  p  <  0  .05 


Table  B-10.  Means  used  for  RMS  Heading  Deviation  (degrees)  Analysis 


INITIAL  APPROACH 

AID 

NOAID  1 

North-Up 

Track-Up 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

17.42 

11.13 

14.28 

14.69 

12.73 

7.49 

7.14 

13.16 

11.69 

1  Overall 

12.46 

12.08 

10.71 

10.50 

13.93 

12.21 

FINAL  APPROACH 

AID 

NOAID  1 

North-Up 

Track-Up 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

1.77 

1.28 

1.53 

1.32 

3.32 

2.32 

2.41 

1.62 

3.66 

2.64 

1  Overall 

2.09 

1.64 

1.87 

1.47 

3.49 

2.48 

ALL  PHASES 

AID 

NOAID 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task  Loading 

9.60 

6.21 

7.90 

7.72 

4.95 

7.51 

5.93 

8.41 

7.17 

6.70 

\  Overall 

7.2B 

6.86 

5.98 

8.71 

7.35 

7.21 
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2.4  RMS  Course  Deviation 


A  2x2x2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  EAP  Orientation  conditions,  2  Approach  Phase  conditions)  was  performed  on 
the  Root  Mean  Square  (RMS)  course  deviation  data.  Because  of  the  potential  for  spurious 
data  in  the  Holding  phase  (due  to  the  numerous  course  changes),  only  the  Initial  and  Final 
Approach  phases  were  included  in  the  analysis.  The  results  of  the  ANOVA  analysis  are 
provided  in  Table  B-1 1  and  the  means  used  in  this  analysis  are  presented  in  Table  B-12. 

A  significant  main  effect  was  found  for  Approach  Phase  (F  =  155.95,  p  <  0.001).  RMS 
course  deviation  was  smaller  in  the  Final  Approach  phase  (mean  =  .092  dots)  than  in  the 
Initial  Approach  phase  (mean  =  2.84  dots).  This  result  is  not  surprising  given  that  the 
navigation  aid  for  the  final  approach  gives  more  accurate  and  sensitive  course  information 
than  the  navigational  aid  for  the  initial  approach.  No  other  significant  main  effects  or 
interactions  were  found. 


Table  B-1 1.  ANOVA  Results  for  RMS  Course  Deviation 


Tested  Effect 

ANOVA  Results  I 

Decision  Aid 

Task  Loading 

EAP  Orientation 

F(l,15)=  0.84,  p  =  . 373 

Approach  Phase 

F(l, 15)  =  755.95,  p  =  . 000* 

Decision  Aid  x  Task  Loading 

F(l,15)=  0.03,  p  =  . 858 

Decision  Aid  x  EAP  Orientation 

Decision  Aid  x  Approach  Phase 

lillll  M II  ITTHnniM 

Task  Loading  x  EAP  Orientation 

F(l,15)=  0.31,  p  =  . 585 

Task  Loading  x  Approach  Phase 

F(l,15)=  0.01,  p  =  . 927 

Approach  Phase  x  EAP  Orientation 

F(l,15)=  0.79,  p  =  . 387 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F91,15)=  0.00,  p  =  . 961 

Decision  Aid  x  Task  Loading  x  Approach  Phase 

Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(l,15)=  0.38,  p  =  . 548 

Decision  Aid  x  Task  Loading  x  EAP  Orientation  x  Approach  Phase 

F(l,15)=  0.00,  p  =  . 969 

*  indicates  statistical  significance,  <  0.05 
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Table  B-12.  Means  Used for  RMS  Course  Deviation  (dots)  Analysis 


INITIAL  APPROACH 

AID 

1  NOAID  1 

North-Up 

Track-Up 

Overall 

IHSSS55!!9 

Track-Up 

Overall 

Low  Task 

2.57 

2.30 

2.63 

2.89 

2.76 

2.23 

2.36 

2.30 

2.97 

2.84 

2.91 

1  Overall 

2.13 

2.46 

2.30 

2.80 

2.87 

2.84 

FINAL  APPROACH 

AID  1 

NOAID  1 

North-Up 

Track-Up 

Overall 

North-Up 

Track-Up 

Overall 

Low  Task 

.051 

mmm\ 

.052 

.061 

.124 

.117 

.121 

.111 

.143 

.127 

1  Overall 

.097 

.084 

.091 

.082 

.102 

.092 

ALL  PHASES 

AID 

NOAID  i 

Overall 

North-Up 

Track- 

Overall 

North- 

Track-Up 

1.04 

1.27 

1.16 

1.48 

1.28 

1.18 

1.21 

1.49 

1.36 

1  Overall 

1.11 

1.26 

1.19 

1.40 

1.48 

1.44 

1.32 

2.5  RMS  Glideslope  Deviation 

A  2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  EAP  Orientation 
conditions)  was  performed  on  the  Root  Mean  Square  (RMS)  course  deviation  data.  Task 
Loading  and  Approach  Phase  were  not  included  in  the  analysis  because  glideslope  data  could 
only  be  obtained  on  the  Low  Task  Loading  precision  approach  conditions  and  during  the 
Final  Approach  phase.  The  results  are  provided  in  Table  B-13  and  the  means  used  in  the 
analysis  are  provided  in  Table  B-14.  No  significant  main  effects  or  interactions  were  found 
in  the  analysis  of  the  glideslope  data. 
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Table  B-13.  ANOVA  Results  for  RMS  Glideslope  Deviation 


Tested  Effect 

ANOVA  Results 

Decision  Aid 

F(l,15)=  0.00,  p  =  . 952 

EAP  Orientation 

F(l,15)=  0.09,  p  =  . 767 

Decision  Aid  x  EAP  Orientation 

F(l,15)=  3.21,  p  =  . 094 

Table  B-14.  Means  Used  for  RMS  Glideslope  Deviation  Analysis 


RMS  Glideslope  Deviation  (dots) 

AID 

NOAID 

Overall 

North-Up  Orientation 

.339 

.559 

.449 

Track-Up  Orientation 

.517 

.290 

.403 

Overall 

.428 

.425 

.426 

3.  Workload  Data 

A  2x2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  EAP  Orientation)  was  performed  on  the  workload  ratings  gathered  with  the 
Subjective  WORkload  Dominance  (SWORD)  technique.  The  results  are  provided  in  Table 
B-13.  The  means  for  this  analysis  are  provided  in  Table  B-14. 


Table  B-13.  ANOVA  Results  for  Workload  Ratings 


Tested  Effect 

ANOVA  Result 

Decision  Aid 

F(l, 15)  =  42.03,  p=. 000* 

Task  Loading 

F(l, 15)  =  33.73,  p  =  . 000* 

EAP  Orientation 

F(l,15)=  1.16,p  =  .299 

Decision  Aid  x  Task  Loading 

F(l,15)  =  61.96,  p=  .000* 

Decision  Aid  x  EAP  Orientation 

F(l,15)=  0.09,  p  =  . 774 

Task  Loading  x  EAP  Orientation 

F(l,15)=  0.26,  p  =  . 618 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F(3,33)=  1.16;p  =  .298 

*  indicates  statistical  significance,  p  <  0.05 


Table  B-14.  Means  Used  for  Workload  Analysis 
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RMS  SWORD  Workload  Ratings 

AID 

NOAID 

Overall 

North-Up 

■BSSl 

Low  Task  Loading 

.056 

.042 

.097 

.089 

.071 

High  Task  Loading 

.084 

.281 

.255 

.018 

Overall 

.063 

.172 

Significant  main  effects  were  found  for  Deeision  Aid  (F  =  42.035,  p  <  0.001)  and 
Approach  Phase  (F  =  33.73,  p  <  0.001);  and  a  significant  interaction  was  found  between 
Deeision  Aid  and  Task  Loading  eonditions  (F  =  6\.96,p  <  0.001).  No  other  signifieant  main 
effects  or  interactions  were  found.  One-way  ANOVAs  were  performed  to  analyze  the  nature 
of  the  signifieant  interaction. 

For  the  Deeision  Aid  by  Task  Loading  interaction,  significant  simple  main  effects  were 
found  for  Decision  Aid  in  both  Low  Task  Loading  (F  =  42.03,  j??  <  .000)  p  =  0.003)  and  High 
Task  Loading  conditions  (F  =  56.03,  p  <  0.001).  The  difference  in  mean  workload  ratings 
between  the  Low  Tasking  Loading  and  High  Task  Loading  in  the  NoAid  condition,  however, 
was  approximately  4  times  larger  (.268  -  .093)  than  in  the  Aid  condition  (.090  -  .049).  Thus, 
the  significant  interaction  would  indicate  that  although  the  Aid  eondition  redueed  workload 
in  both  Low  Task  Loading  and  High  Task  Loading  conditions,  the  Aid  eondition  was  more 
effective  in  reducing  workload  in  the  High  Task  Loading  eondition.  The  means  that  were 
used  in  this  analysis  are  presented  in  Table  B-15. 


Table  B-15.  Means  Used  in  the  Workload  Analysis 


Decision  Aid  by  Task  Loading  Interaction 

Aid 

NoAid 

Low  Task  Loading 

.049 

.093 

High  Task  Loading 

.090 

.268 

4.  Situational  Awareness  Data 

A  2x2x2  Repeated  Measure  ANOVA  (2  Decision  Aid  conditions,  2  Task  Loading 
conditions,  2  EAP  Orientation)  was  performed  on  the  situational  awareness  ratings  gathered 
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with  the  Subjective  WORkload  Dominance  (SWORD)  technique.  The  results  showed 
significant  main  effects  for  the  Decision  Aid  (F  =  32.89,  p  <  .001)  and  Task  Loading  (F  = 
25.31, <  .001)  conditions.  No  other  significant  main  effects  or  interactions  were  found. 
The  results  are  provided  in  Table  B-16  and  the  means  are  provided  in  Table  B-17. 


Table  B-16.  ANOVA  Results  for  Situational  Awareness  Ratings 


Tested  Effect 

ANOVA  Result 

Decision  Aid 

F(l,15)=  32.89,  p  =  .000* 

Task  Loading 

F(l,15)=  25.31,  p=. 000* 

EAP  Orientation 

F(l,15)=  0.15,  p=. 705 

Decision  Aid  x  Task  Loading 

F(l,15)=  1.98,  p=, 180 

Decision  Aid  x  EAP  Orientation 

F(l,15)=  0.30,  p  =  . 594 

Task  Loading  x  EAP  Orientation 

F(l,15)=  0.57,  p  =  . 461 

Decision  Aid  x  Task  Loading  x  EAP  Orientation 

F(3,33)  =  0.40,  p  =  .535 

*  indicates  statistical  significance,  p  <  0.05 


Table  B-1 7.  Means  Used  for  Situational  Awareness  Ratings 


RMS  SWORD  Workload  Ratings 

AID 

NOAID 

Overall 

North-Up 

Track-Up 

North-Up 

Track-Up 

Low  Task  Loading 

.237 

.215 

.106 

.101 

.165 

High  Task  Loading 

.127 

.127 

.042 

.044 

.041 

Overall 

.182 

.171 

.074 

.073 

.103 
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APPENDIX  C  -  SUMMARY  OF  QUESTIONNAIRE  RESPONSES 


Introduction 


The  questionnaires  used  in  the  APES  evaluation  are  provided  in  Appendix  A.  This 
Appendix  summarizes  questionnaire  responses  for  15  of  the  16  pilots  who  participated  in 
this  study.  (Final  questionnaire  ratings  were  not  obtained  for  one  pilot.)  Where 
appropriate,  the  responses  are  summarized  in  tables. 


SESSION  QUESTIONNAIRE  RESPONSES 


The  table  below  provides  mean  ratings  for  ease  of  task  performance.  These  ratings  were 
obtained  after  each  approach  test  condition. 

Ease  of  Task  Performance  Scale: 

5  =  A.  Extremely  Easy 
4  =  B.  Moderately  Easy 
3  =  C.  Neutral 
2  =  D.  Moderately  Difficult 
1  =  E.  Extremely  Difficult 


AID 

NOAID 

OVERALL 

HIGH  TASK  LOAD 

TRK-UP 

TRK-UP 

TRK-UP 

NTH-UP 

Assess  A/C  Performance 

IQ9 

4.06 

miijm 

m&B 

3.38 

3.74 

Fly  Holding  Pattern 

4,44 

3.58 

Fly  Initial  Approach 

4.50 

4.21 

4.25 

n^mi 

3.46 

3.63 

Fly  Final  Approach 

4.46 

3.63 

4.31 

m^ii 

3.31 

3.04 

4.13 

4.06 

3.58 

mm 

4.05 

4.45 

4.36 

3.88 

mm 

mEfli 

Overall  Mean 

4.28 

4.35 

3.83 

3.83 

4.22 

4.17 

3.46 

3.37 

FINAL  QUESTIONNAIRE  RESPONSES 


SECTION  1.  QUALITY  OF  APES  “PROCEDURAL”  ADVICE 

The  tables  on  the  following  pages  provide  the  means  and  the  frequencies  of  the  acceptability 
ratings  used  to  evaluate  APES  performance.  Pilot  comments  are  provided  below  each  table. 

Acceptability  Scale: 

5  =  A.  Completely  acceptable:  No  changes  needed;  acceptable  as  is. 

4  =  B.  Moderately  acceptable:  Minor  changes  desirable;  effectiveness  not  impaired. 

3  =  C.  Borderline:  Changes  desirable;  effectiveness  may  be  impaired. 

2  =  D.  Moderately  unacceptable:  Changes  required,  effectiveness  may  be  impaired. 

1  =  E.  Completely  unacceptable:  Redesigned  required,  effectiveness  severely  impaired. 


HEADING  PROMPTS  | 

MEAN 

FREQUENCY  1 

5 

4 

3 

2 

1 

Holding 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness 

4.60 

10 

4 

1 

0 

0 

Consistency 

4.67 

11 

3 

1 

0 

0 

Initial  Approach 

Logic 

4.67 

11 

3 

1 

0 

0 

Timeliness 

4.80 

12 

3 

0 

0 

0 

Consistency 

4.93 

14 

1 

0 

0 

0 

Final  Approach 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness 

4,67 

10 

5 

0 

0 

0 

Consistency 

4.93 

14 

1 

0 

0 

0 

Missed  Approach 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness' 

4.33 

7 

6 

2 

0 

0 

Consistency 

4.87 

13 

2 

0 

0 

0 

Overall 

Logic 

4.73 

11 

4 

0 

0 

0 

Timeliness 

4.47 

7 

8 

0 

0 

0 

Consistency! 

4.73 

11 

4 

0 

0 

0 

COMMENTS: 


Subject  1: 
altitude. 

Subject  2: 

Subject  3: 

Subject  7: 

Subject  8: 


At  one  missed  approach,  the  computer  told  me  to  turn  prior  to  reaching  the  designated  turn 

Final  approach  track  -  some  calls  seem  early,  some  late. 

Missed  approach  prompts  are  late. 

If  you  are  heading  to  a  fix,  she  needs  to  ensure  that  the  Navaid,  radial/course  is  set. 

Need  immediate  MAP  instructions. 


Subject  9:  Announce  the  missed  approach  point  with  “and  or  go-around”  call  or  something. 


Subject  10:  Holding  timeliness  -  was  late  to  recognize  overshoots.  Consistency  -  sometimes  gave 
numerous  small  corrections  in  holding  that  were  not  significant  enough  to  voice  prompt  -  Let  the  bank 
steering  bar  silently  guide  the  pilot  more. 

Subject  1 1 :  One  of  the  heading  prompts  placed  me  inside  an  arc,  then  the  voice  reminded  me  to  stay  on  the 
arc!  Missed  approach  instructions  seemed  rushed.  Might  be  better  not  to  even  have  them. 

Subject  13:  Missed  approach  turn  was  given  early  on  every  approach.  Lead  turn  for  Final  Approach 
consistently  early. 

Subject  14:  Need  to  release  large  heading  changes  coming  to  lAF  or  holding  fix.  Will  promote  chasing  as 
is. 

Subject  16:  There  was  a  couple  of  times  when  I  would  have  teardropped  but  the  D.A.  recommended  a  turn 
to  parallel. 


1  COURSE  PROMPTS 

MEAN 

FREQUENCY 

5 

4 

3 

2 

1 

Holding 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness 

4.93 

14 

1 

0 

0 

0 

Consistency 

4.93 

14 

1 

0 

0 

0 

Initial  Approach 

Logic 

4.67 

11 

3 

1 

0 

0 

Timeliness 

4.80 

12 

3 

0 

0 

0 

Consistency 

5.00 

15 

0 

0 

0 

0 

Final  Approach 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness 

4.93 

14 

1 

0 

0 

0 

Consistency  ^ 

5.00 

15 

0 

0 

0 

0 

Missed  Approach 

Logic 

4.80 

12 

3 

0 

0 

0 

Timeliness 

4.73 

12 

2 

1 

0 

0 

Consistency 

5.00 

15 

0 

0 

0 

0 

Overall 

Logic 

4.73 

11 

4 

0 

0 

0 

Timeliness 

4.87 

13 

2 

0 

0 

0 

Consistency 

5.00 

15 

0 

0 

0 

0 

COMMENTS: 

Subject  3:  (Missed  approach  prompts)  are  too  late  to  be  safe. 

Subject  4:  Course  prompts  ignored  on  large  intercepts  until  initial  roll-out  completed. 

Subject  7:  Just  telling  me  to  maintain  an  arc  when  a  deviation  occurs  is  not  really  adequate  -  help  me  make 
a  correction,  i.e.,  “turn  10°  left  to  intercept  12  DME  ARC.” 

Subject  8:  Need  immediate  MAP  prompt.  Should  direct  “maintain _ are”  during  initial  turn. 

Should  /  Could  direct  next  course  input  sooner. 
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AIRSPEED  PROMPTS  I 

MEAN 

FREQUENCY 

5 

4 

3 

2 

1 

Holding 

Logic 

4.60 

11 

2 

2 

0 

0 

Timeliness 

4.40 

9 

4 

1 

1 

0 

Consistency 

4.60 

11 

2 

2 

0 

0 

Initial  Approach 

Logic 

4.60 

11 

2 

2 

0 

0 

Timeliness 

4.33 

8 

5 

1 

1 

0 

Consistency 

4.60 

11 

2 

2 

0 

0 

Final  Approach 

Logic 

4.53 

10 

3 

2 

0 

0 

Timeliness 

4.40 

9 

4 

1 

1 

0 

Consistency 

4.60 

11 

2 

2 

0 

0 

Missed  Approach 

Logic 

4.60 

11 

2 

2 

0 

0 

Timeliness 

4.33 

8 

5 

1 

1 

0 

Consistency 

4.60 

11 

2  1 

2 

0 

0 

Overall 

Logic 

4.53 

10 

3 

2 

0 

0 

Timeliness 

4.40 

9 

4 

1 

1 

0 

Consistency 

4.60 

11 

2 

2 

0 

0 

COMMENTS: 

Subject  1 :  When  making  the  turn  to  final,  I  would  like  to  start  slowing  to  approach  speed  prior  to  localizer 
intercept. 

Subject  2:  Call  for  slower  airspeed  earlier.  Get  setup  sooner  so  as  to  be  stabilized  on  profile  earlier. 

Subject  3:  (Missed  approach  prompts)  are  too  late  to  be  safe. 

Subject  5:  Airspeed  prompts  were  too  sensitive.  The  airspeed  warnings  should  be  pilot  selectable. 

Subject  7:  Tolerance  should  be  -0/+15  -  you  should  never  be  slow  without  a  prompt.  15  kt  fast  should  be 
allowed  for  gusts. 

Subject  9:  Airspeed  prompts  fit  this  scenario  but  it  greatly  simplified  from  the  way  airspeed  is  dealt  with  in 
the  planes  I’ve  flown. 

Subject  10:  Overall  the  airspeed  prompt  tolerances  are  too  tight  and  some  (missed  approach)  aren’t  even 
necessary.  Probably  just  appropriate  for  Final,  and  then  +15,  -  10  knots. 

Subject  1 1 :  Missed  approach  instructions  seemed  rushed.  Might  be  better  not  to  even  have  them. 

Subject  14:  Simply  call  airspeed  -  don’t  say  maintain  250  knots. 


ALTITUDE  PROMPTS 

MEAN 

FREQUENCY 

5 

4 

3 

2 

1 

Holding 

Logic 

4.60 

11 

2 

2 

0 

0 

Timeliness 

4.73 

13 

1 

0 

1 

0 

Consistency 

4.80 

13 

1 

1 

0 

0 

Initial  Approach 

Logic 

4.67 

12 

1 

2 

0 

0 

Timeliness 

4.73 

13 

1 

0 

1 

0 

Consistency 

4.80 

13 

1 

1 

0 

0 

Final  Approach 

Logic 

4.20 

9 

2 

2 

2 

0 

Timeliness 

4.47 

11 

2 

0 

2 

0 

Consistency 

4.80 

13 

1 

1 

0 

0 

Missed  Approach 

Logic 

4.47 

11 

1 

2 

1 

0 

Timeliness 

4.40 

10 

3 

0 

2 

0 

Consistency 

4.60 

12 

1 

1 

1 

0 

Overall 

Logic 

4.53 

11 

1 

3 

0 

0 

Timeliness 

4.53 

11 

2 

1 

'  1 

0 

Consistency 

4.73 

12 

2 

1 

0 

_ Q_ 

COMMENTS: 

Subject  3:  (Missed  approach  prompts)  are  too  late  to  be  safe. 

Subject  5:  Altitude  prompts  were  too  sensitive.  The  airspeed  warnings  should  be  pilot  selectable. 

Subject  7:  Tolerance  should  be  -0ft/+200ft  for  mandatory  or  at-or-above,  +0ft/“100ft  for  at  or  below. 
Verbage  should  be  “climb  to  . . . “descend  to  . . . .  “  or  “maintain  . . . .”  as  appropriate 

Subject  8:  “Descend  to  DH  of _ recommended  versus  “maintain _ 

Subject  9:  The  aid  given  seems  real  busy  at  the  lAF/FAF  and  MAP  points.  Maybe  some  of  the  info  can  be 
spread  out  more. 

Subject  10:  Too  many  missed  approach  altitude  prompts  (about  4)  during  climbout.  Voice  prompt  once 
and  then  visual  display  is  enough. 

Subject  1 1 :  This  was  very  useful  to  me.  Missed  approach  instructions  seemed  rushed.  Might  be  better  not 
to  even  have  them. 

Subject  13:  When  flying  an  ILS  &  intercepting  G.S.  above  a  step  down  fix  altitude,  a  warning  is 
unnecessarily  given.  Often,  altitudes  depicted  are  min  altitudes  and  not  mandatory.  “Minimum  altitude 
2600  feet”  might  be  better  than  “descend  to  . . . 

Subject  14:  Same  as  airspeed  -  simply  call. 

Subject  15:  On  almost  every  missed  approach,  the  altitude  prompt  was  given  several  times  within  a  short 
time  span.  When  descending  to  the  ILS  DH,  the  prompt  should  say  “Descend  to  480  feet”  instead  of 
maintain  480  feet  When  climbing  on  missed  approach,  the  prompt  should  say  “climb  to  2500  feet.” 


RADIO  CHANNEL/FREQUENCY  PROMPTS  I 

MEAN 

FREQUENCY 

5 

4 

3 

2 

1 

Holding 

Logic 

4.86 

13 

0 

1 

0 

0 

Timeliness 

4.93 

13 

1 

0 

0 

0 

Consistency 

5.00 

14 

0 

0 

0 

0 

Initial  Approach 

Logic 

4.93 

14 

1 

0 

0 

0 

Timeliness 

4.87 

13 

2 

0 

0 

0 

Consistency 

5.00 

15 

0 

0 

0 

0 

Final  Approach 

Logic 

4.93 

14 

1 

0 

0 

0 

Timeliness 

4.80 

12 

3 

0 

0 

0 

Consistency 

4.93 

14 

1 

0 

0 

0 

Missed  Approach 

Logic 

4.93 

14 

1 

0 

0 

0 

Timeliness 

4.80 

12 

3 

0 

0 

0 

Consistency 

4.93 

14 

1 

0 

0 

0 

Overall 

Logic 

4.80 

13 

1 

1 

0 

0 

Timeliness 

4.73 

11 

4 

0 

0 

0 

Consistency 

4.93 

14 

_ i_ 

0 

0 

0 

COMMENTS: 

Subject  1:  During  holding,  while  using  TACAN  for  navigation.  I  would  like  to  set  the  ILS  frequency  in  my 
VOR  receiver.  The  decision  aid  did  not  like  this  idea. 

Subject  2:  I  felt  that  when  NAVAID  changes  were  required,  TACAN  to  ILS,  they  could  have  happened 
earlier.  You  can  fly  the  course  you  are  on  using  the  head  or  tail  of  the  needle. 

Subject  3:  (Missed  approach  prompts)  are  too  late  to  be  safe. 

Subject  8:  Could  go  to  some  NAVAIDs  sooner  when  done  with  last  one. 

Subject  10:  Excellent  aid  -  should  also  include  mode  select  prompt  from  TAC  to  ILS  on  Final  and  vice 
Versa  on  Missed  Approach. 

Subject  1 1 :  This  was  the  most  useful  prompt  of  all  with  some  of  multiple  changes  required  on  some  of  the 
approaches. 

Subject  13:  The  automatic  selection  is  outstanding  (with  pilot  consent  required  before  entering). 

Subject  15  I  didn’t  look  to  see  what  altitude  the  prompts  came  on  during  the  missed  approach,  but  they 
should  wait  until  the  aircraft  is  above  1000  feet  AGL. 


SECTION  IL  QUALITY  OF  APES  “DEVIATION”  ADVICE 


The  table  below  depicts  the  means  and  the  frequencies  of  the  Acceptability  Ratings  used  to 
evaluate  APES  performance  Pilots  comments  are  also  provided. 

Acceptability  Scale: 

5  =  A.  Completely  acceptable:  No  changes  needed;  acceptable  as  is. 

4  =  B.  Moderately  acceptable:  Minor  changes  desirable;  effectiveness  not  impaired. 

3  =  C.  Borderline:  Changes  desirable;  effectiveness  may  be  impaired. 

2  =  D.  Moderately  unacceptable:  Changes  required,  effectiveness  may  be  impaired. 

1  =  E.  Completely  unacceptable:  Redesigned  required,  effectiveness  severely  impaired. 


DEVIATION  PROMPTS  1 

MEAN 

FREQUENCY  1 

. -5 

4 

3 

2 

1 

Holding 

Logic 

4.40 

8 

5 

2 

0 

0 

Timeliness 

4,47 

10 

3 

1 

1 

0 

Consistency 

4.53 

10 

3 

2 

0 

0 

Initial  Approach 

Logic 

4.67 

11 

3 

1 

0 

0 

Timeliness 

4.67 

12 

2 

0 

1 

0 

Consistency 

4.73 

12 

2 

1 

0 

0 

Final  Approach 

Logic 

4.20 

7 

4 

4 

0 

0 

Timeliness 

4.47 

10 

3 

1 

1 

0 

Consistency ! 

4.60 

11 

2 

2 

0 

0 

Missed  Approach 

Logic 

4.47 

9 

4 

2 

0 

0 

Timeliness 

4.67 

12 

2 

0 

1 

0 

Consistency 

4.73 

12 

2 

1 

0 

0 

Overall 

_ Logic 

4.43 

8 

6 

1 

0 

0 

Timeliness 

4.50 

10 

4 

0 

I  1 

0 

Consistency 

4.63 

11 

3 

_ 1 _ 

1 _ Q_ 

_ 0— 

COMMENTS: 

Subject  1:  When  the  gust  model  is  active,  keeping  an  airspeed  within  10  knots  is  nearly  impossible,  yet  the 
decision  aid  still  gives  deviation  warnings. 

Subject  2:  Would  suggest  adding  the  direction  you  need  to  go.  (i.e.,  “go  up,”  “turn  left,”  “slow  down,” 
“come  right”  etc. 

Subject  3:  Aggravating,  but  necessary  and  challenging. 

Subject  5:  Airspeed  and  altitude  prompts  were  too  sensitive.  The  airspeed  warnings  should  be  pilot 
selectable. 

Subject  6:  Should  have  a  feature  that  turns  off  voice  prompt  as  soon  as  you  hit  the  “enter”  button. 

Example:  Go  of  airspeed  by  10  kts.  You  realize  it  right  away  and  then  the  voice  prompts  tell  you.  You  hit 
the  “enter’  button  but  the  voice  has  to  say  the  entire  prompt,  you  can’t  turn  it  off. 

Subject  7 :  See  previous  comments  regarding  tolerances. 
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Subject  8:  I  really  liked  these! 

Subject  9:  Heading  deviations  of  less  the  than  5  degrees  carried  for  a  long  can  result  in  large  track 
deviations.  For  holding  entries  of  teardrops  maybe  the  tolerance  should  be  tighter.  Also  holding  teardrop 
heading  to  the  full  DME  distance  can  cause  nav  errors.  Maybe  hold  teardrop  heading  for  1-1  1/2  minutes 
then  parallel  course.  Give  100  above^elow  call  instead  of  maintain  altitude.  Give  airspeed  as  “slow”  or 
“fast”  instead  of  maintain  airspeed.  Or  “slow  to  250”)  (increase  to  250) . 

Subject  10:  Too  many  minor  deviation  voice  prompts.  Remember  other  systems  have  beepers,  bells, 
alarms,  buzzers,  whistles,  plus  there’s  radio  and  intercom  messages  to  hear.  All  together  we  are  beginning 
to  overload  pilots  with  talking  airplanes. 

Subject  13:  Altitude  warnings,  as  previously  stated,  may  be  given  when  the  altitude  depicted  is  a  minimum 
only  and  your  are  intentionally  high. 

Subject  14:  Too  often,  and  too  much  sensitivity  -  need  to  better  capture  logic. 

Subject  15:  The  tolerance  bands  at  which  the  prompts  “come  on”  should  be  pilot  selectable  (to  a  point)  in  a 
real  aircraft. 

Subject  16:  A  couple  of  times  the  verbal  and  written  heading  and  altitude  prompts  appeared  different  by  up 
to  10  degrees  or  100  feet  (may  just  be  pilot  perception) 
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SECTION  III.  TASK  ACCOMPLISHMENT  AND  EAP  ORIENTATION 

The  table  on  the  following  page  provide  means  and  frequencies  of  pilot  response  ratings 
to  the  statement: 

The  Decision  Aid  _ my  ability  to . 

Rating  Scale: 

5  =  A.  Substantially  Enhanced 
4  =  B.  Moderately  Enhanced 
3  =  C.  Did  Not  Affect 
2  =  D.  Moderately  Degraded 
1  =  E.  Substantially  Degraded 


C-12 


a)  identify  and  navigate  to  the  holding  fix. 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation 

Overall  _ 

b)  determine  direction  of  turn  into  holding. 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ _ 

Overall _ 

c)  identify  and  navigate  to  the  lAF. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall _ 

d)  identify  approach  radials  /  arc. _ 

Track-Up  EAP  Orientation  _ 

North-Up  EAP  Orientation 

Overall _ 

e)  identify  approach  altitude. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall _ 

f)  identify  and  navigate  to  the  FAF. _ 

Track-Up  EAP  Orientation  _ 

North-Up  EAP  Orientation _ 

Overall _ 

g)  identify  the  DH  or  MPA. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall  _ _ 

h)  identify  and  navigate  to  the  MAP. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall  _ _ _ 

i)  identify  and  set  Navaids.  _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall  _ 

j)  determine  intercept  headings  /  angles. 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall  _ 

k)  determine  lead  points  for  arcs  /  courses. 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall  _ 

l)  determine  airspeeds. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall _ 

m)  determine  a/c's  position  relative  to  fixes. 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ 

Overall _ 

n)  develop  a  mental  picture. _ 

Track-Up  EAP  Orientation _ 

North-Up  EAP  Orientation _ _ 

Overall 


MEAN 

FREOUET 

5 

4 

3 

4.40 

7 

7 

1 

4.13 

4 

9 

2 

4.47 

7 

8 

0 

4,20 

8 

2 

5 

3.87 

4 

5 

6 

4.33 

9 

2 

4 

4.40 

8 

5 

2 

4.13 

6 

5 

4 

4.47 

9 

4 

1  2 

4.33 

8 

4 

1  3 

4.38 

4 

6 

5 

4.33 

8 

4 

3 

4.27 

6 

7 

i  2 

4.24 

6 

6 

3 

4.53 

8 

7 

0 

4.20 

6 

6 

3 

3,93 

4 

6 

5 

4.27 

7 

5 

3 

3.67 

4 

4 

5 

4.14 

3 

5 

5 

3.93 

6 

4 

3 

4.20 

8 

3 

3 

4.07 

5 

4 

5 

4.20 

8 

3 

3 

4.07 

7 

3 

4 

4.00 

6 

4 

4 

4.27 

8 

4 

2 

4.31 

7 

6 

2 

3.98 

4 

7 

4 

4.31 

7 

6 

2 

3.98 

3 

9 

3 

3.71 

2 

7 

6 

3.98 

3 

9 

3 

3.27 

1 

2 

12 

3.94 

2 

1 

12 

3.47 

3 

1 

11 

4.27 

9 

1 

5 

3.91 

6  1 

2 

7 

4.27 

9 

1 

5 

4.40 

10 

1 

4 

4.07 

7 

2 

6 

4.40 

10 

1 

4 
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COMMENTS: 


A)  IDENTIFY  AND  NAVIGATE  TO  THE  HOLDING  FIX. 

Subject  13:  Heading  for  Fix  to  Fix  nav  is  outstanding. 

B)  DETERMINE  DIRECTION  OF  TURN  INTO  HEADING 
Subject  2:  Both  modes  definitely  assisted  in  this  area. 

Subject  16:  EAP  provided  necessary  picture. 

E)  IDENTIFY  APPROACH  ALTITUDE 

Subject  1 1 :  Some  of  the  altitudes  of  some  approach  plates  was  difficult  to  see. 

F)  IDENTIFY  AND  NAVIGATE  TO  THE  FAF 

Subject  2:  For  all  of  these  seeing  a/c  and  any  fix  and  that  relationship  is  invaluable.  This  comment  applies 
to  all  these  categories. 

G)  IDENTIFY  THE  DECISION  HEIGHT  OR  MINIMUM  DESCENT  ALTITUDE 
Subject  7:  The  type  of  alt  should  be  identified,  i.e.,  “at  decision  height.” 

Subject  10:  Don’t  say  “maintain”  when  prompting  DH  altitude.  Call  it  “DH”  or  “MDA”  at  the  voice 
prompt. 

Subject  13:  Maybe  use  “Decision  Height  646”  instead  of  “descend  to  ...  “  for  Precision  Approach. 
Subject  16:  The  text  was  too  small  on  the  EAP  -  D.A.  helped. 

H)  IDENTIFY  AND  NAVIGATE  TO  THE  MAP 
Subject  7:  Could  add  “at  the  missed  approach  point.” 

Subject  8:  Should  cue  MAP. 

Subject  1 1 :  Comments  seemed  to  rapid. 

Subject  16:  Good  verbal  cues  -  basically  a  no-brainer. 

I)  IDENTIFY  AND  SET  NAV  AIDS 

Subject  I :  Some  of  the  frequency  /  channels  are  difficult  to  read. 

Subject  1 1 :  Very  good. 

J)  DETERMINE  INTERCEPT  HEADINGS  /  ANGLES 
Subject  9:  Logic  seemed  a  bit  off. 

Subject  13:  Slightly  slow  on  course  correction  Headings  (?)  early  lead  points. 

K)  DETERMINE  LEAD  POINTS  FOR  ARCS  /  COURSES 
Subject  5:  More  could  be  done  here. 

Subject  8:  Could  use  slightly  larger  turn  radius. 

Subject  9:  Logic  seemed  a  bit  off 

Subject  16:  Takes  away  the  mental  gymnastics. 

L)  DETERMINE  AIRSPEEDS 
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Subject  10:  Let  pilots  pick  speeds. 


M)  DETERMINE  AIRCRAFT’S  POSITION  RELATIVE  TO  APPROACH  FIXES 
Subject  15:  I  don’t  recall  the  decision  aid  telling  me  my  position  relative  to  approach  fix. 
Subject  16:  EAP  was  enough. 

N)  DEVELOP  A  MENTAL  PICTURE  FOR  ANTICIPATING  FUTURE  EVENTS 
Subject  15:  The  EAP  performed  this  function,  not  the  decision  aid. 
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SECTION  IV.  PILOT-VEHICLE  INTERFACE 


The  table  below  shows  the  means  and  the  frequencies  of  the  Acceptability  Ratings  used  to 
evaluate  the  Pilot- Vehicle  Interface.  Pilot  comments  are  also  provided. 

Acceptability  Scale: 

5  =  A.  Completely  acceptable:  No  changes  needed;  acceptable  as  is. 

4  =  B.  Moderately  acceptable:  Minor  changes  desirable;  effectiveness  not  impaired. 

3  =  C.  Borderline:  Changes  desirable;  effectiveness  may  be  impaired. 

2  =  D.  Moderately  unacceptable:  Changes  required,  effectiveness  may  be  impaired. 

1  =  E.  Completely  unacceptable:  Redesigned  required,  effectiveness  severely  impaired. 


PILOT- VEHICLE  INTERFACE 

MEAN 

FREQUENCY 

5 

4 

3 

2 

1 

1.  Heading  Prompts 

4.33 

8 

6 

0 

0 

1 

2.  Attitude  Prompts 

4.00 

6 

6 

1 

1 

1 

3.  Airspeed  Prompts 

4.07 

5 

8 

1 

0 

1 

4.  Course  Prompts 

4.67 

11 

3 

1 

0 

0 

5.  Freq/Chan  Prompts 

4.73 

12 

2 

1 

0 

0 

6.  Audio 

a.  Voice  Quality 

4.67 

11 

3 

1 

0 

0 

b.  Understandability 

4.67 

12 

1 

2 

0 

0 

c.  Phraseology 

3.93 

5 

7 

1 

1 

1 

d.  Sequence 

4.53 

9 

5 

1 

0 

0 

7.  Visual 

a.  Readability 

4.70 

7  : 

3 

0 

0 

0 

b.  Understandability 

4.70 

7 

3 

0 

0 

0 

c.  Phraseology 

4.60 

6 

4 

0 

0 

0 

d.  Sequence 

4.60 

6 

4 

0 

0 

0 

e.  Location 

3.91 

4 

4 

1 

2 

0 

COMMENTS: 

1.  HEADING  PROMPTS 


Subject  5:  Either  make  pilot  selectable,  re-evaluate  windows  (e.g.,  percentage  of  altitude  vs.  +/-  100  feet), 
or  leave  off  of  design. 

Subject  9:  Tolerance  window  (+/-  5  degrees)  situation  dependent. 

Subject  13:  Tolerance  slightly  high. 


2,  ALTITUDE  PROMPTS 


Subject  3:  Tolerance  Window  (+/-  100  feet)  -  too  small  for  my  fists. 

Subject  5:  Either  make  pilot  selectable,  re-evaluate  windows  (e.g.,  percentage  of  altitude  vs.  +/-  100  feet), 
or  leave  off  of  design. 

Subject  7:  Tolerance  Window  should  be  -0/+200ft. 
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Subject  10:  Audio  Display:  too  many  minor  deviation  prompts.  Too  many  repeats  on  climb  to/descent 
Too  tight  -‘‘ok,  ok  -  Pm  going. 

Subject  13:  Tolerance  slightly  high. 

3.  AIRSPEED  PROMPTS 
Subject  1:  Perhaps  too  frequent. 

Subject  5:  Either  make  pilot  selectable,  re-evaluate  windows  (e.g.,  percentage  of  altitude  vs.  +/-  100  feet), 
or  leave  off  of  design. 

Subject  6:  Audio  Display:  Would  be  able  to  turn  it  off  if  you  know  you’re  at  altitude  or  airspeed. 

Subject  7:  Tolerance  Window  should  be  -0/+15kts. 

Subject  10:  OK  on  Final,  but  too  (+/-10  kts)  tight  for  approach  maneuvering. 

Subject  11:  In  windy  conditions,  this  was  a  little  distracting. 

4.  COURSE  PROMPTS 

Subject  10:  Tolerance  window  of  +/-  2  degrees  too  tight  for  approach  maneuvering. 

ADDITIONAL  COMMENTS 

Subject  9:  Tolerance  windows  need  to  be  situation  dependent  with  certain  defaults  and  maybe  pilot 
programmable  to  other  windows. 

Subject  14:  Need  to  give  the  tasks  and  put  the  next  event  in  line  to  accommodate  different  situations  and 
pilot  foresight  levels. 

7.  WAYS  VOICE  INTERFACE  COULD  BE  IMPROVED 

Subject  2:  Add  direction  to  correct. 

Subject  3:  Seems  OK  to  me. 

Subject  4:  Instead  of  “Maintain _ on  approach  segments  -  could  use  -  “descend  and  maintain.” 

Subject  5:  Either  make  (prompts)  pilot  selectable,  re-evaluate  windows  (e.g.,  percentage  of  altitude  vs.  +/- 
100  feet),  or  leave  off  of  design. 

Subject  6:  As  stated  before,  be  able  to  turn  it  off. 

Subject  7:  Verbage:  when  directing  altitudes  specify  the  type,  i.e.,  descend  to  decision  height,  600ft,  or 
maintain  at  or  above  two  thousand  five  hundred. 

Subject  8:  Phraseology  of  Voice  Message:  DH  callout;  change  terminology. 

Subject  9:  Different  phrasing  for  altitude/airspeed  deviations  (10  knots  fast/slow,  100  above,  below)  Put 
climb  or  descend  before  the  no.  of  feet  above/below  call. 

Subject  10:  Maintain  altitude  “xyz”  when  its  the  DH,  we  never  “maintain  DH  -  potential  safety  issue  here. 
Subject  11:  A  little  more  excitement  would  help! 

Subject  13:  Use  of  “decision  height”  and  “minimum  attitude”  would  be  helpful. 

Subject  14:  Need  better  speech  synthesizer,  pause  longer  between  phrases. 
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Subject  15:  Phraseology  of  Voice  Message:  When  descending  to  the  ILS  DH,  the  prompt  should  say 
“descend  to  “480  feet  ”  . . .  When  climbing  on  missed  approach,  the  prompt  should  say  “climb  to  2500 
feet.”  ....  Voice  should  be  less  “mechanical.” 

Subject  16:  Sequence  of  Voice  Message  Format:  Sometimes  the  multiple  info  phrases  came  too  close 
together. 


9.  WAYS  TEXT  MESSAGE  DISPLAY  COULD  BE  IMPROVED 


Subject  1 :  I  found  I  hardly  ever  looked  at  the  message  display. 

Subject  2:  They  were  find,  but  honestly  I  only  looked  at  them  maybe  twice.  What  was  displayed  was 
acceptable.  I  feel  that  you  really  didn’t  need  that  screen. 

Subject  3:  Only  viewed  twice-  well  out  of  visual  window  for  me. 

Subject  5:  No  comment  - 1  never  used  the  CRT.  I  actually  think  that  it  is  unnecessary. 

Subject  7:  Phraseology  of  text  messages  -  abbreviate  more.  I  didn’t  really  use  it. 

Subject  8:  Didn’t  even  use  it.  Not  required.  I  think  it  is  of  little  use  (ex.  occasional  cross  reference  if  bug 
does  appear  on  gauges. 

Subject  9:  Didn’t  really  use  them  because  they  were  so  far  out  of  the  cross-check  region. 

Subject  10:  Make  it  more  obvious  what  Navaids  are  selected.  Overall  the  CRT  is  the  best  feature  of  the 
system.  The  old  saying  “  a  picture  is  worth  a  thousand  words  really  applies  here.  Thus  the  project  should 
focus  heavily  on  the  visual  and  cut  much  of  the  voice  stuff.  I  think  that  if  you  have  to  pick  one  track  -  then 
track-up  is  the  way  to  go.  But,  you’ll  need  to  add  one  feature  -  Allow  slewing  the  displayed  aircraft  position 
so  I  can  see  what’s  behind  me  when  I’m  heading  outbound.  The  bottom  line  is  that  I  know  my  relative 
position  to  the  airfield  and  navaids.  I  don’t  care  about  the  North  Pole  -  except  in  December! 

Subject  13:  Rarely  used  text  messages,  cockpit  space,  and  pilot  workload/interpretation,  may  limit 
usefulness  of  textual  messages. 

Subject  14:  Didn’t  use  text  messages  hardly  at  all.  Not  necessary. 

Subject  15:  I  never  looked  at  it. 

Subject  16:  Instead  of  highlighting  the  text  box  and  giving  me  an  altitude  above  (author  comment:  scratch 
pad),  use  this  format  (author  comment:  as  in  the  textual  window) 
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SECTION  V.  GENERAL  QUESTIONS 


The  table  below  shows  the  frequencies  of  pilot  ratings  to  general  questions.  The  rating  scale 
is  for  each  question  is  presented  in  the  right  column.  Pilot  comments  to  these  and  opemend 
questions  are  provided. 


QUESTION 

FREQUENCY 

RATING  SCALE 

5 

4 

3 

2 

1 

1.  Follow  Procedural  Advice 

4 

11 

0 

0 

0 

5=Always,  4=Often,  3=Sometimes, 

2= Almost  Never,  l=Never 

2.  Follow  Deviation  Advice 

7 

7 

1 

0 

0 

5=Always,  4=Often,  3=Sometimes, 

2= Almost  Never,  l=Never 

3.  EAP  Orientation  Affect 
Decision  Aid  Usage 

n/a 

n/a 

n/a 

4 

10 

2=Yes,  1-No 

4.  Wind/Rvw  affect  on  Workload 

0 

0 

10 

5 

0 

3-SubstantiaIly  Increased,  2=  Moderately 
Increased,  1-Did  Not  Affect 

5.  Wind/Rvw  affect  on  DecAid 

6 

3 

6 

0 

0 

5-Substan  Increase,  4-Mod  increase  3-No 
Affect,  2-Mod  Decrease,  1-Subtant  Decrease 

6.  Decision  Aid  Improve  Safety 

7  : 

8 

0 

0 

^  0 

5-Strongly  Agree,  4-Mod  Agree,  3-Neutral, 
2-Mod  Disagree,  1 -Strong  Disagree 

7.  Overall  Usefulness  of  Decision 
Aid 

10 

5 

0 

0 

0 

5-Extreme  Benefit,  4-Mod  Benefit, 

3-Neutral,  2=Mod  Hinder,  1 -Extreme  Hinder 

8.  Other  Conditions 

n/a 

n/a 

6 

7 

2 

3-Yes,  2-Not  Sure,  l=No 

9.  Develop  a  Strategy  Using 
Decision  Aid 

n/a 

n/a 

2 

13 

n/a 

3-Yes,  2=No 

COMMENTS: 

1.  FREQUENCY  OF  FOLLOWING  PROCEDURAL  ADVICE 

Subject  1 :  Sometimes  I  would  choose  my  own  headings  when  I  didn’t  like  the  one  the  computer  gave  me. 
Subject  2:  Most  of  the  time  the  prompts  were  necessary. 

Subject  3:  Some  tum/descend  messages  occur  before  the  fix  -  they  are  then  a  prompt  vs.  a  command. 
Subject  5:  Always,  some  number  of  exceptions. 

Subject  1 1 :  Most  of  the  time  it  was  accurate. 

2.  FREQUENCY  OF  FOLLOWING  DEVIATION  ADVICE 

Subject  2:  Most  of  the  time  the  prompts  were  necessary 

Subject  15:  Was  usually  already  correcting  back  to  the  desired  parameters. 

3.  EAP  ORIENTATION  EFFECT  ON  DECISION  AID  USE 

Subject  4:  Track-Up  -  picture  great  help  in  Viewing  (?)  Aids  directions. 

Subject  5:  Either  way  is  satisfactory. 

Subject  10:  There  are  pros  and  cons  to  each.  Give  the  pilot  the  option  to  select  track  type  as  desired. 
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Subject  1 1 :  Track-Up  caused  some  portion  of  the  displays  to  be  hidden  during  parts  of  the  approach.  The 
decision  aid  helped  most  in  these  situations. 

Subject  13:  Track-Up  orients  correctly  with  HSI,  less  confusing. 

Subject  14:  Provided  both  confidence  in  answer  and  safety  vent  for  not  following  DA. 

Subject  16:  Not  consciously. 

4.  EFFECT  OF  THE  COMBINATION  OF  WIND  GUSTS  AND  NO  PRIOR  REVIEW  OF  THE 
APPROACH  ON  WORKLOAD 

Subject  1 :  The  wind  gusts  more  than  the  “no  prior  review”  increased  workload,  because  it  was  easy  to 
review  the  approach  enroute  to  the  holding  fix. 

Subject  2:  Required  more  concentration  to  “avoid”  the  prompts  which  is  good. 

Subject  3 :  Approaches  became  a  handful  w/winds  and  no  review  -  do-able  but  much  more  work. 

Subject  4:  Made  it  necessary  to  concentrate  alot  on  aircraft  control  -  to  the  extent  that  I  ignored  keeping  up 
with  AID  (i.e.,  hit  “enter”  tot  get  ffeq  settings) 

Subject  5:  The  wind  gusts  were  more  severe  than  reality  (by  a  factor  of  102). 

Subject  16:  Wind  model  was  bogus!  Airplane  does  not  cause  that  many  deviations.  Also  no  review  is  not 
a  big  deal  except  that  part  of  an  approach  calls  for  an  approach  plate  review. 

5.  WIND  GUST  AND  NO  PRIOR  REVIEW  OF  THE  APPROACH  PLATE  EFFECT  ON  THE  USE  OF 
THE  DECISION  AID 

Subject  2:  Did  not  rely  on  it  any  more,  just  made  me  work  hard. 

Subject  4:  Last  two  runs  -  approach  would  have  been  impossible  without  it. 

Subject  5:  Not  enough  confidence  in  the  D.A.  yet  to  let  it  tell  me  what  to  do. 

Subject  15:  I  used  the  Decision  Aid  regardless  of  wind  condition. 

Subject  16:  This  is  only  due  to  increased  amount  of  deviations  caused  by  bogus  wind  model. 

6.  DECISION  AID  IMPROVES  SAFETY  WHEN  FLYING  INSTRUMENT  APPROACHES 
No  Comments 

7.  OVERALL  UTILITY  OF  THE  DECISION  AID  FOR  ASSISTING  IN  PERFORMING  INSTRUMENT 
APPROACHES 

Subject  2:  Anyone  assisting  someone  shooting  an  instrument  approach  is  always  welcome. 

Subject  8:  With  improvements  as  system  matures. 

Subject  11:1  think  it  would  have  a  tendency  to  make  people  lazy  about  studying  approaches  before  flying 
them. 

Subject  13:  Moderately  beneficial  if  the  pilot  doesn’t  miss  or  forget  anything.  Potential  for  avoiding  large 
deviations  warrants  the  system  as  extremely  beneficial. 

8.  OTHER  CONDITIONS  WHERE  THIS  DECISION  AID  MIGHT  BE  USEFUL? 

Subject  1:  Departures 

Subject  2:  Navy  aircraft  carrier  landings.  This  system  adapted  to  that  environment  would  be  great. 

Subject  9:  Low  level  routes  and  airdrop  operations.  Air  refueling. 

Subject  11:  Pop-up  to  dive  bomb  attacks. 
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Subject  15:  Aircraft  emergencies  . . .  Display  ore  read  the  checklist  to  you  (especially  in  a  single-seat 
fighter). 

Subject  16:  Couple  it  to  the  autopilot  and  let  the  pilot  monitor  the  approach  in  very  bad  weather. 

9.  STRATEGY  WHEN  USING  DECISION  AID 

Subject  4:  Used  it  for  Fix  to  Fix  -  incredibly  “natural”  and  easy  to  adapt  to.  Also  arcs  and  approach 
segments. 

Subject  7:  Accepted  “scratch  pad”  data  all  at  once,  then  confirmed  it  using  the  voice  prompts. 

Subject  13:  Experimented  with  using  instruments  conventionally,  expanding  the  depiction  and  using  it  for 
deviations,  etc. 

10.  WHAT  DID  YOU  LIKE  BEST  ABOUT  THE  DECISION  AID? 

Subject  1 :  Its  ability  to  monitor  HDG/A/S/ALT  and  automatic  setting  of  courses  and  NAVAID  frequency. 
Subject  2:  During  the  times  when  I  wasn’t  paying  any  attention,  it  was  real  nice  knowing  it  was. 

Subject  3:  Track-Up  increased  SA  immeasurably. 

Subject  4:  The  MAP. 

Subject  6:  Great  tool  when  there  was  no  review. 

Subject  5:  The  visual  display  of  lAP  with  the  aircraft  tracking  throughout. 

Subject  7:  Auto  input  of  prompts  /freq/heading,/course/alt. 

Subject  8:  Corrections,  initial  headings. 

Subject  9:  Hearing  what  to  do  next  at  the  right  time.  Prompting  and  timeliness. 

Subject  10:  Again  the  visual  display  was  great,  (from  his  other  comments,  I  think  subject  was  referring  too 
EAP) 

Subject  1 1 :  The  reminders  of  frequency./  channel  changes  on  busy  approaches. 

Subject  13:  Automatic  selection  of  frequency  and  courses,  and  the  overall  safety  aspect. 

Subject  14:  1)  Rapid  selection  of  the  point  to  point  problems.  2)  Back-up  on  altitude  deviation. 

Subject  15:  The  NAVAID  frequency  and  course/heading  changes  . . .  just  press  “enter”  and  everything  is 
set  for  you  ...  a  big  workload  reducer. 

Subject  16:  Altitude  prompting  -  This  is  easy  to  miss  in  the  jet. 

1 1 .  WHAT  DID  YOU  LIKE  LEAST  ABOUT  THE  DECISION  AID? 

Subject  1:  Its  authoritarian  altitude. 

Subject  2:  Not  much  at  all. 

Subject  3:  Late  missed  approach  prompts. 

Subject  4:  Prompts  sometimes  wrong. 

Subject  5:  The  altitude/airspeed  prompts. 

Subject  6:  Voice  prompts  were  excessive  at  times. 

Subject?:  Text  Monitor  display 
Subject  8:  MAP 

Subject  9:  Long  streams  of  instruction  without  a  pause  at  lAF/FAF. 

Subject  10:  The  nuisance  voice  inputs  -  “airplane  -  be  quiet!” 

Subject  1 1 :  The  hurried  nature  of  missed  approaches. 

Subject  13:  Frequent  distracting  altitude  warnings  approaching  decision  height. 
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Subject  14:  It’s  single-mindedness.  When  I  chose  an  alternative  outside  the  D.A.,  it  wouldn’t  help  much. 
Subject  15:  The  mechanical  sound  of  the  voice. 

Subject  16:  1)  Verbal  cues  sometimes  too  close  together.  2)  The  way  the  textual  info  is  presented.  You 
tend  to  just  hit  “enter”  without  interpreting  the  info. 

12.  ADDITIONAL  COMMENTS 

Subject  2:  This  system  should  be  considered  for  carrier  use  if  not  already. 

Subject  3:  Possible  to  become  reliant/complacent  while  using  the  decision  aid. 

Subject  4:  On  large  intercepts  -  maybe  turn  rate  and  winds  could  be  taken  into  account  as  well  as  (?) 
objective. 

Subject  9:  Maybe  some  way  to  delay  certain  comments  until  previous  commands  acknowledged. 

Subject  15:  Gotta  have  the  EAP  with  or  without  the  decision  aid. 
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APPENDIX  D  -  REPRESENTATIVE  PROCESS  FLOW 
CHARTS  FOR  EXPERT  SYSTEM  DEVELOPMENT 
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Aircraft-Specific  Tasks 


I 

(Return) 


Aircraft-Specific  Tasks 
for  Missed  Approach  (MA) 

r 

\  Aircraft-  / 

Process  Flow 

Retrieve  Missed  Approach  Airspeed 

_ A  Specific  / 

(2.4) 

1 

_ 1 

\  Database/ 

T 

(Return) 


J 
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Approach-Specific  Tasks 
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Approach-Specific  Tasks 
Process  Flow  (Cont.) 


Approach-Specific  Tasks 
for  Final  Approach 
Process  Flow 


Retrieve  Final  Approach  Data  (from  Database) 

a)  Navaid  Required  for  Final  Approach 

b)  Final  Approach  Fix  (FAF) 

c)  Final  Approach  Fix  Altitude 

d)  Minimum  Descent  Altitude/Descent  Height 

e)  Final  Approach  Course _ 


Approach 
Specific 
^Database  > 


Is  FAF  Navam 
set  properl)^ 


Set  lA  Navaid  to  DB  value  and  notify 
pilot  of  change. 


Approach-Specific  Tasks 
for  Missed  Approach 
Process  Flow 


Retrieve  Missed  Approach  Data  (from  Database) 

a)  Navaid  Required  for  Missed  Approach  (MA) 

b)  Missed  Approach  Point  (MAP) 

c)  Outbound  Course/Track 

d)  Missed  Approach  Altitude 


Approach 
Specific 
^Database  ^ 


Is  MANavaic 
^t  properly^ 


Set  MA  Navaid  to  DB  value  and  notify 
pilot  of  change. 


^Return! 


Holding 

Generic  Tasks 

Overall  Process  Flow  (5) 


t 


Holding  Example 
Generic  Subtasks 
Process  Flows  -  (5) 


Determine  Holding 
Pattern  Timing 


Determine  Bank  Angle 
Correcting  for  Drift 


Determine  Direction  of 
Turn 


Direction  of  Turn  = 
Direction  of  Holding  Pattern 


Initial  Approach 

GenericTasks 

Overall  Process  Flow  -  (6) 


(Tell  pilot  to)  Adjust  airspeed  to  Inbound  Value 


Determine  descent  point  inbound 


Current  Airspeed  > 
Gear  Limit  Airspeed^ 


(Tell  pilot  to)  Reduce  airspeed  below  gear  limit  speed 


Current  Altitude  -  Desired  Altitude 
<=  1000  feet  7^ — 


Tell  pilot  that  a/c  is  within  1000ft  of  desired  altitude 


Tell  pilot  to  level  of  at  100  feet  above  desired  altitude 


Final  Approach 
Generic  Tasks 
Overall  Process  Flow  -  (7) 


D-9 


Missed  Approach 
Generic  Tasks 
Overall  Process  Flow  -  (8) 


